Display apparatus, display control method, and display system

ABSTRACT

A display apparatus includes an image acquisition unit, an image extraction unit, a registration unit, a display control unit, a coordinate generation unit, and a motion detection unit. The coordinate generation unit generates, based on a detection result of a detection unit configured to detect the position of an object in a three-dimensional space, coordinates of the object in a screen. The motion detection unit detects a motion of the object based the coordinates in the screen generated by the coordinate generation unit. The display control unit displays a first image on the screen. When the motion is detected by the motion detection unit, the display control unit further displays a second image on the screen based on coordinates corresponding to the detected motion, and changes the display of the first image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT international application Ser.No. PCT/JP2016/057043 filed on Mar. 7, 2016 which designates the UnitedStates, incorporated herein by reference, and which claims the benefitof priority from Japanese Patent Applications No. 2015-058418, filed onMar. 20, 2015, incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

Embodiments relate to a display apparatus, a display control method, anda display system.

2. Description of the Related Art

Recent performance improvements to computer devices have enabled imagesto be easily displayed by computer graphics using three-dimensionalcoordinates (hereinafter abbreviated as “3DCG”). In 3DCG, it is a commonpractice to set a regular or random motion to each object arranged in athree-dimensional coordinate space so that the objects are displayed asa moving image. In the moving image, the objects can be represented asif they moved independently in the three-dimensional coordinate space.

Japanese Patent No. 4161325 discloses a technology of giving a motion toan image created by an animation creator by handwriting on an image by3DCG and combining the image with a background image prepared in advanceto be displayed.

A technology that enables a user image created by an ordinary user byhandwriting to be easily displayed in a screen by 3DCG has been soughtafter. In the case where a user image is displayed in a screen by 3DCG,it is preferred that the motion of an object, such as the body of auser, in a three-dimensional space be reflected to operation of the userimage displayed in the screen, because the level of amusement is furtherimproved. Conventionally, however, it is difficult to reflect the motionof the body of a user to the operation of a user image displayed in ascreen.

SUMMARY OF THE INVENTION

According to an embodiment, a display apparatus includes an imageacquisition unit, an image extraction unit, a registration unit, adisplay control unit, a coordinate generation unit, and a motiondetection unit. The image acquisition unit is configured to acquire animage including a drawing region drawn by a user. The image extractionunit is configured to extract, from the acquired image, a first imagebeing an image in the drawing region. The registration unit isconfigured to register attribute information indicating attributes thatis set with respect to the extracted first image and is used forcontrolling of moving the first image on a screen. The display controlunit is configured to control display on the screen. The coordinategeneration unit is configured to generate, based on a detection resultof a detection unit configured to detect a position of an object in athree-dimensional space, coordinates of the object in the screen. Themotion detection unit is configured to detect a motion of the objectbased on the coordinates. And, the display control unit is configured tofurther display, when the motion is detected by the motion detectionunit, a second image on the screen based on the coordinatescorresponding to the detected motion, and change the display of thefirst image to which attribute information of a certain attribute amongthe attributes of the first image displayed on the screen is set.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an example of adisplay system according to a first embodiment;

FIG. 2 is a diagram illustrating an example of a sheet that can beapplied to the first embodiment;

FIG. 3 is a diagram illustrating a display example of a medium accordingto the first embodiment;

FIG. 4 is a diagram illustrating an example of a sheet set on an imageacquisition device;

FIG. 5 is a diagram illustrating an example of an image projected on aprojection medium according to the first embodiment;

FIG. 6 is a diagram illustrating an example of an image in which a userobject is further added according to the first embodiment;

FIG. 7 is a diagram illustrating an example of an image in which an iconimage is displayed according to the first embodiment;

FIG. 8 is a diagram illustrating an example where a motion fordisplaying an icon image on an image is performed according to the firstembodiment;

FIG. 9 is a diagram illustrating another example of emphasized displayaccording to the first embodiment;

FIG. 10 is a diagram illustrating an example where an icon imageindicating a screenshot is displayed at a position of motion coordinatesaccording to the first embodiment;

FIG. 11A is a diagram for describing an example of an imageconfiguration according to the first embodiment;

FIG. 11B is a diagram for describing an example of the imageconfiguration according to the first embodiment;

FIG. 12 is a block diagram illustrating a configuration of an example ofa PC that can be applied in the first embodiment;

FIG. 13 is a function block diagram illustrating an example of functionsof the PC that can be applied in the first embodiment;

FIG. 14 is an exemplary function block diagram illustrating the functionof an image analysis unit according to the first embodiment;

FIG. 15 is an exemplary function block diagram illustrating the functionof a display control unit according to the first embodiment;

FIG. 16 is an exemplary flowchart schematically illustrating processingfor displaying an image acquired by an image acquisition deviceaccording to the first embodiment;

FIG. 17 is an exemplary flowchart schematically illustrating processingperformed when an identification image is acquired by an identificationimage reading device according to the first embodiment;

FIG. 18 is an exemplary flowchart illustrating the operation of an imageanalysis unit according to the first embodiment;

FIG. 19 is a diagram for describing processing for extracting each imagefrom a sheet according to the first embodiment;

FIG. 20 is an exemplary flowchart illustrating processing forregistering identification information, attribute information, and auser image to a user image DB according to the first embodiment;

FIG. 21 is a diagram for describing how to generate managementinformation by associating attribute information and a user image withidentification information;

FIG. 22 is a diagram illustrating an example of a display screenindicating a folder structure of a user image DB according to the firstembodiment;

FIG. 23 is a diagram illustrating an example of the configuration of afile in which attribute information is stored according to the firstembodiment;

FIG. 24A is a diagram illustrating an example of a detection range of asensor according to the first embodiment;

FIG. 24B is a diagram illustrating an example of a detection range of asensor according to the first embodiment;

FIG. 25 is a diagram illustrating an example of coordinates in an imageof an object that are determined by a method according to the firstembodiment;

FIG. 26 is a diagram illustrating an example of setting of regions in animage according to the first embodiment;

FIG. 27 is a diagram illustrating only an attribute setting regionarranged on a sheet according to the first embodiment;

FIG. 28 is a diagram illustrating an example of association between anattribute and a region according to the first embodiment;

FIG. 29 is a diagram illustrating an example of association between anattribute and an icon image according to the first embodiment;

FIG. 30 is an exemplary flowchart illustrating motion determinationprocessing according to the first embodiment.

FIG. 31A is a diagram illustrating an example of the motion of a userobject in an image when it is determined that contact is absentaccording to the first embodiment;

FIG. 31B is a diagram illustrating an example of the motion of a userobject in an image when it is determined that contact is absentaccording to the first embodiment;

FIG. 32 is an exemplary flowchart illustrating motion determinationprocessing according to a modification of the first embodiment;

FIG. 33 is a diagram illustrating an exemplary configuration of adisplay system according to a second embodiment;

FIG. 34 is a diagram for describing how to control the movement of auser object in correspondence with the position of a microphoneaccording to the second embodiment;

FIG. 35 is an exemplary function block diagram illustrating the functionof a PC according to the second embodiment;

FIG. 36 is an exemplary function block diagram illustrating the functionof a display control unit according to the second embodiment;

FIG. 37A is a diagram illustrating an exemplary format for a user todesignate a sound pattern according to the second embodiment;

FIG. 37B is a diagram illustrating an exemplary format for a user todesignate a sound pattern according to the second embodiment;

FIG. 38 is an exemplary flowchart schematically illustrating processingfor displaying an image acquired by an image acquisition deviceaccording to the second embodiment;

FIG. 39 is an exemplary flowchart schematically illustrating user imagedisplay processing based on sound patterns according to the secondembodiment;

FIG. 40 is an exemplary flowchart illustrating display processingperformed when a user object for a user image associated with a soundpattern has already been displayed in an image according to the secondembodiment;

FIG. 41 is a diagram illustrating an exemplary configuration of adisplay system according to a first modification of the secondembodiment;

FIG. 42 is an exemplary flowchart schematically illustrating processingaccording to the first modification of the second embodiment; and

FIG. 43 is a diagram illustrating an exemplary configuration of adisplay system in which a PC for operation guidance is providedaccording to a second modification of the second embodiment.

The accompanying drawings are intended to depict exemplary embodimentsof the present invention and should not be interpreted to limit thescope thereof. Identical or similar reference numerals designateidentical or similar components throughout the various drawings.

DESCRIPTION OF THE EMBODIMENTS

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention.

As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

In describing preferred embodiments illustrated in the drawings,specific terminology may be employed for the sake of clarity. However,the disclosure of this patent specification is not intended to belimited to the specific terminology so selected, and it is to beunderstood that each specific element includes all technical equivalentsthat have the same function, operate in a similar manner, and achieve asimilar result.

An object of an embodiment is to enable the motion of an object in athree-dimensional space to be reflected to operation of a user imagedisplayed in a screen. Referring to the accompanying drawings, a displayapparatus, a display control method, and a display system according toembodiments are described in detail below.

Schematic Configuration According to First Embodiment

FIG. 1 illustrates an exemplary configuration of a display systemaccording to a first embodiment. In FIG. 1, a display system 1 aincludes a computer (PC) 10 a, a projector (PJ) 11, a database (DB) 12a, a sensor 13, an image acquisition device 20, and an identificationimage reading device 21. The PC 10 a has an image management programaccording to the first embodiment installed thereon, and controls theoperation of the display system 1 a in accordance with the imagemanagement program. The PJ 11 projects an image 100 onto a projectionmedium 14 in accordance with an image signal output from the PC 10 a.

The sensor 13 detects the position of an object. For example, the sensor13 has a camera incorporated therein, and can detect, on the basis of animage of a target object included in a taken image acquired by thecamera, the distance to the target object and the position of the targetobject. The sensor 13 is placed on the projection surface side of theprojection medium 14. The sensor 13 may be placed on the projectionsurface side of the projection medium 14 at either an upper part or alower part of the projection medium 14 or placed at any of the top,bottom, left or right. Detection results of the sensor 13 are suppliedto the PC 10 a.

For example, Kinect (trademark) by Microsoft Corporation can be used forthe sensor 13. The sensor 13 is not limited to this example, and animaging unit and an image processing unit may be combined to detect theposition of a moving target object by analyzing a taken image by theimage processing unit or to detect the position of a moving targetobject by using infrared rays or ultrasonic waves.

The DB 12 a includes databases, and stores therein, for example, images(image data) to be used for the PC 10 a to project on the projectionmedium 14 by the PJ 11.

The image acquisition device 20 includes, for example, a camera. Theimage acquisition device 20 uses the camera to image a sheet 50 set at apredetermined position, and acquires an image of the sheet 50. In a morespecific example, the image acquisition device 20 includes a camera, animaging stage for placing the sheet 50 thereon, and a jig configured tofix the camera at a predetermined distance and a predetermined positionwith respect to the imaging stage. For example, when the sheet 50 isplaced on the imaging stage, and a predetermined operation is performedon the image acquisition device 20, an image of the sheet 50 is taken bythe camera, and the taken image is output from the camera. The imageacquisition device 20 supplies the acquired image to the PC 10 a. Theimage acquisition device 20 is not limited to the camera, and may be ascanner device configured to scan an image by an image sensor to acquirethe image.

The identification image reading device 21 acquires an image of a medium40, reads an identification image 41 displayed on the medium 40 from theacquired image, and supplies the read identification image 41 to the PC10 a.

A printing medium that is printable, such as paper and a resin film, canbe used as the medium 40. For example, a service provider who provides aservice using the display system 1 a uses a PC 30 to generate anidentification image on the basis of identification information set inaccordance with a predetermined format, and uses a printer 31 to printthe generated identification image on the medium 40. In this case, media40, 40 . . . on which different identification images 41 are printed canbe created.

The medium 40 is not limited to a printing medium. For example, what iscalled a tablet device, which has a central processing unit (CPU) and adata communication function and is capable of displaying an image on athin display integrally formed with a casing, can be used as the medium40. In this case, for example, an identification image transmitted fromthe outside is received by the data communication function, and. inaccordance with the received identification image, the identificationimage is displayed on the display.

For the identification image 41, for example, an image obtained byencoding identification information consisting of a character string canbe used. Examples of the identification image 41 that can be usedinclude a two-dimensional code, such as QR code (trademark). Withoutbeing limited thereto, a character string indicating identificationinformation may be directly printed and used as the identification image41, or a code obtained by converting identification information into animage in a format different from QR code (trademark) may be used.

Identification information used for the identification image 41 can begenerated by the following Expression (1), for example. In Expression(1), addition symbol “+” represents the merge of character strings.identification information=fixed code+time information indicating timeat which identification information is generated   (1)

In Expression (1), the fixed code is, for example, a code determined foreach service provider. Without being limited thereto, the fixed code maybe configured by combining a first fixed code and a second fixed code.In this case, for example, the first fixed code may be used as a codeindicating a provider of the display system 1 a, and the second fixedcode may be used as a code determined for each service provider. In thiscase, the second fixed code may be notified in advance from the PC 30 tothe PC 10 a.

For the time information, time in an environment where theidentification image 41 is generated (for example, time measured by thePC 30) can be used. In one example, when the identification image 41 isformed by being printed on the medium 40, the PC 30 may acquire timeinformation indicating time at which processing of printing theidentification image 41 on the medium 40 is performed. In this case, thetime information is acquired as information of six digits including atleast hour, minute, and second. The time information may include anotherfiner unit, or include units such as day and month.

The identification information is not limited to the example ofExpression (1). For example, identification information may beconfigured by using only time information indicating time at whichidentification information is generated.

On the sheet 50 in this example, a hand-drawing region 43 used for auser 2 to draw a picture by hand and an attribute setting region 42 forsetting attributes for the picture drawn in the hand-drawing region 43are displayed. On the sheet 50, a region in which the medium 40 can beplaced without being overlapped with the attribute setting region 42 andthe hand-drawing region 43 is provided.

FIG. 2 illustrates an example of the sheet 50 that can be applied in thefirst embodiment. In the sheet 50 illustrated in FIG. 2, thehand-drawing region 43 used to draw a picture by hand, the attributesetting region 42 for setting attributes for the picture drawn in thehand-drawing region 43, and a placement region 44 for placing the medium40 therein are arranged. In the example in FIG. 2, the placement region44 is displayed with a border to make it easier for a user to recognize.In the example in FIG. 2, the attribute setting region 42 can set sixattributes: “walk”, “go forward”, “fly”, “plant”, “building”, and“swim”.

Markers 51 ₁; 51 ₂, and 51 ₃ are arranged at three of the four cornersof the sheet 50. Detecting the markers 51 ₁ , 51 ₂, and 51 ₃ from anoriginal image obtained by acquiring an image of the sheet 50 with theimage acquisition device 20 enables the orientation and the size of thesheet 50 to be known.

FIG. 3 illustrates a display example of the medium 40 according to thefirst embodiment. As illustrated in FIG. 3, an identification image 41is displayed on the medium 40. In the example in FIG. 3, theidentification image 41 is displayed at substantially the center of themedium 40, but without being limited to this example, the identificationimage 41 can be displayed at a desired position on the medium 40. In thefirst embodiment, the identification image 41 only needs to be displayedon the medium 40. However, another information (such as logo image) maybe displayed on the medium 40 together with the identification image 41.

Schematic Operation of Display System According to First Embodiment

Next, the operation of the display system 1 a according to the firstembodiment is schematically described with reference to FIG. 1. Fordescription, the display system 1 a is installed in a venue of an eventhosted by a service provider. The service provider uses the PC 30 andthe printer 31 in advance or in the venue to prepare media 40, 40, . . .on which identification images 41 based on different pieces ofidentification information are printed. The service provider alsoprepares sheets 50, 50, . . . .

A user 2 who has visited the venue receives the medium 40 and the sheet50 from the service provider. The user 2 draws a picture in thehand-drawing region 43 on the sheet 50, and sets attributes in theattribute setting region 42 for the drawn picture. The user 2 sets thesheet 50 on the image acquisition device 20, and places the medium 40 inthe placement region 44 on the sheet 50 such that the identificationimage 41 s displayed on the front surface side.

FIG. 4 illustrates an example of the sheet 50 set on the imageacquisition device 20. In FIG. 4 a picture 45 is drawn in thehand-drawing region 43 on the sheet 50. The picture 45 is not limited tothe one directly drawn on the sheet 50, and a seal having a picture 45drawn thereon in advance may be attached or placed, or an image createdby another PC may be printed in the hand-drawing region 43 on the sheet50.

In the example in FIG. 4, in the attribute setting region 42, the secondframe on the upper row for the attribute “go forward” is filled out, andit is understood that the attribute “go forward” has been set for thepicture 45. Furthermore, the medium 40 is placed in the placement region44.

In this manner, in the state in which the sheet 50 is set -with respectto the image acquisition device 20, for example, the user 2 performs anoperation of instructing the image acquisition device 20 to acquire animage of the sheet 50. The image acquisition device 20 images the sheet50 in response to the operation, and outputs the image of the sheet 50acquired by the imaging. The image output from the image acquisitiondevice 20 is supplied to the PC 10 a.

The PC 10 a analyzes the image supplied from the image acquisitiondevice 20 to acquire an identification image 41 included in the image,an image in the attribute setting region 42, and an image in thehand-drawing region 43. The PC 10 a analyzes the acquired identificationimage 41 to extract identification information from the identificationimage 41. The PC 10 a analyzes the image in the attribute setting region42 to acquire the set attribute. The PC 10 a extracts an image of apicture 45 from the hand-drawing region 43. The image of the picture 45is hereinafter referred to as “user image”.

The PC 10 a stores the user image and the attribute acquired from theimage on the sheet 50 in the DB 12 a in association with theidentification information extracted from the image on the sheet 50. Atthe same time, the PC 10 a transmits the user image to the projector 11so that the user image is projected on the projection medium 14. In thiscase, the PC 10 a provides the user image with parameters for a motionand coordinates on the basis of the attribute. The PC 10 a furtherprovides the user image with parameters for a motion and coordinates onthe basis of the user image itself.

By drawing different pictures 45 on sheets 50 and repeating theabove-mentioned processing following the image acquisition of the sheet50 by the image acquisition device 20 on each sheet 50, the user 2 candisplay user images on the projection medium 14. In this case, anidentification image 41 is acquired from the medium 40 placed on thesheet 50, and hence the identification image 41 common to the differentsheets 50 can be acquired. Thus, common identification information canbe associated with user images.

Display Example of User Image

Next, an example of display of user images that can be applied in commonto embodiments is described. FIG. 5 illustrates an example of an imageprojected on the projection medium 14 according to the first embodiment.In FIG. 5, an image 100 is projected on the projection medium 14. Inthis example, the image 100 includes a background image 110 and an image(fixed object) 111 formed of a fixed object.

In the example in FIG. 5, the background, image 110 includes a skyregion 110 a and a land region 110 b, and the land region 110 b is animage having a depth from the lower end of the image 100 toward aboundary between the land region 110 b and the sky region 110 a.Specifically, the image 100 has coordinate axes (x axis, y axis)indicating the positions in the horizontal direction and the verticaldirection, respectively, and a z axis that is a coordinate axisindicating the position in the depth direction. Thus, the position ofeach image displayed in the image 100 is indicated by coordinates (x, y,z) consisting of the three coordinate axes (x axis, y axis, z axis).

In FIG. 5 an icon image 112 displayed at the upper left corner of theimage 100 is arranged in order to acquire a screenshot of the image 100.

The image 100 can further display images 120 ₁ to 120 ₄ based on userimages. Images based on the user images that are displayed in the image100 are hereinafter referred to as “user objects”. In this example, theuser objects 120 ₁ to 120 ₄ are associated with common identificationinformation. The user objects 120 ₂ and 120 ₃ are displayed in the image100 such that the positions thereof are fixed on the basis of theattribute acquired from the attribute setting region 42. On the otherhand, the user objects 120 ₁ and 120 ₄ are displayed in the image 100while moving in the land region 110 b at predetermined speeds on thebasis of the attribute.

As illustrated in FIG. 6, another user object can be added to theabove-mentioned state in which the user objects 120 ₁ to 120 ₄ aredisplayed on the image 100. The image 100 in FIG. 6 is obtained byadding user objects 120 ₁₀ to 120 ₁₃ to the image 100 in FIG. 5.

For example, a user (referred to as “second user”) different from theuser (referred to as “first user”) who displayed the above-mentioneduser objects 120 ₁ to 120 ₄ uses a medium 40 on which identificationinformation 41 different from the identification information 41displayed on the medium 40 carried by the first user is displayed tocause the image acquisition device 20 to acquire images of sheets 50,50, . . . on each of which the picture 45 is drawn. In this manner, theuser objects 120 ₁₀ to 120 ₁₃ based on the respective user images aredisplayed in the image 100, and each user image and its attribute areadded and stored in the DB 12 a in association with the identificationinformation 41.

In the example in FIG. 6, the user object 120 ₁₀ is displayed in theimage 100 while moving in the sky region 110 a at a predetermined speedon the basis of the attribute acquired from the attribute setting region42 on the sheet 50. The user objects 120 ₁₂ and 120 ₁₃ are displayed inthe image 100 while moving in the land region 110 b at predeterminedspeeds on the basis of the attribute. The user object 120 ₁₁ isdisplayed in the image 100 such that the position thereof is fixed onthe basis of the attribute. In FIG. 6, the user objects 120 ₁ and 120 ₄are displayed in the image 100 such that the positions thereof havemoved from the state in FIG. 5.

In the above description, the image acquisition device 20 is used toacquire an image of the sheet 50, and an image based on a user imageincluded in the sheet 50 is displayed in the image 100. In the firstembodiment, an image based on a user image stored in the DB 12 a can befurther displayed in the image 100.

In one example, identification information and user images correspondingto the user objects 120 ₁₀ to 120 ₁₃ and their attributes are stored inthe DB 12 a in association with one another by the above-mentionedsecond user. As exemplified in FIG. 5, the user objects 120 ₁ to 120 ₄by the first user have already been displayed in the image 100.

In this state, the second user places, on the identification imagereading device 21, the medium 40 that was used to acquire each of theimages corresponding to the user objects 120 ₁₀ to 120 ₁₃ from thesheets 50, and instructs the identification image reading device 21 toread an identification image 41. The identification image reading device21 reads the identification image 41 on the medium 40 in accordance withthe instruction, and transmits the read identification image 41 to thePC 10 a.

The PC 10 a analyzes the identification image 41 transmitted from theidentification image reading device 21 to acquire identificationinformation. The PC 10 a searches the DB 12 a on the basis of theacquired identification information to acquire a user image and anattribute associated with the acquired identification information. ThePC 10 a displays the user image thus acquired from the DB 12 a in theimage 100 in accordance with the attribute. In this manner, similarly toFIG. 6, the user objects 120 ₁₀ to 120 ₁₃ by the second user are addedto the user objects 120 ₁ to 120 ₄ by the first user in the image 100.

As described above, by holding an identification image 41 used to storea user image in the DB 12 a, the second user can read a user imagecreated in the past from the DB 12 a and display the user image in theimage 100.

Specifically, the display system 1 a according to the first embodimentis configured such that an image of a medium 40 on which anidentification image 41 is displayed is placed on a sheet 50, and theimage of the sheet 50 including the medium 40 is acquired, thereby beingcapable of managing a user image based on a picture provided by a userin association with identification information indicated by theidentification image 41. The identification image 41 is displayed on themedium 40, which is independent of the sheet 50, and hence the user cansuccessively apply the medium 40 to sheets 50 such that a large numberof pictures can be managed as user images in groups by usingidentification information.

Operation Example Depending on User Motion

Next, a change of display in response to the motion of a user that canbe applied in common to the embodiments is described. In the firstembodiment, the sensor 13 can detect some motion of a user during thedisplay of the image 100, and the state of the image 100 and the stateof each user object in the image 100 can be changed.

For example, the user performs an operation of moving the position of anobject within a detection range of the sensor 13, such as throwing outthe arm to the front or swinging the arm, in front of the projectionmedium 14 on which the image 100 is being displayed. This operation isreferred to as “motion”. The sensor 13 detects the motion, that is,detects an object in the detection range, and outputs positionalinformation indicating the position at which the motion is detected. Thepositional information is supplied to the PC 10 a. The PC 10 a generatescoordinates in the image 100 on the basis of the positional informationsupplied from the sensor 13. In the following, the coordinates arereferred to as “motion coordinates”. The PC 10 a can classify the motioninto types of actions on the basis of a detection result of the sensor13.

In the DB 12 a, images to be displayed on the image 100 in response tothe motion of the user are stored in advance. Unless otherwisedescribed, the image is hereinafter referred to as “icon image”. Forexample, the icon image is stored in the DB 12 a in association witheach region in the image 100. For example, the sky region 110 a and theland region 110 b described above with reference to FIG. 5 can beapplied to regions in the image 100. The regions are not limited to thesky region 110 a and the land region 110 b, and another region such as awater region corresponding to a waterfront may be further added. Theicon image can be associated with regions.

The PC 10 a determines whether which of the regions in the image 100 thegenerated motion coordinates are included in. On the basis of thedetermination result, the PC 10 a reads an icon image associated withthe region in which the motion coordinates are included from the DB 12a. The PC 10 a displays the icon image read from the DB 12 a at aposition indicated by the motion coordinates in the image 100.

FIG. 7 illustrates an example of the image 100 in which the icon imagesare displayed in the manner described above. In the example in FIG. 7,icon images 130 ₁ and 130 ₂ are displayed at the upper left corner andin the vicinity of the center in the image 100, respectively.Specifically, an icon image 130 ₁ associated with the sky region 110 ais displayed in response to the motion of the user at the upper leftcorner of the image 100. Similarly, an icon image 130 ₂ associated withthe land region 110 b is displayed in response to the motion of the usernear the center of the image 100.

The icon images 130 ₁ and 130 ₂ are displayed preferentially over thebackground image 110, the fixed object 111, and each of the user objects120 ₁ to 120 ₄ and 120 ₁₀ to 120 ₁₃.

When icon images are associated with a region in which motioncoordinates are included, the PC 10 a randomly selects one icon imagefrom among the icon images in accordance with a predetermined motion ofthe user, and displays the selected icon image in the image 100. Whenthe user wants to switch the displayed icon image to another icon image,the user makes a motion again at the same position. The PC 10 a randomlyselects one icon image from the icon images associated with the region,deletes the currently displayed icon image, and displays the selectedicon image. In this case, the number of times by which the user makes apredetermined motion at the same position may be measured, and only theoperation of deleting the icon may be performed every predeterminednumber of times. For another example, the icon image may be moved whenthe motion is performed continuously for a given period or longer.

The state of a user object that has already been displayed on the image100 can be changed depending on the display of an icon image on theimage 100. The change in state of the user object depending on thedisplay of the icon image is described with reference to FIG. 8 and FIG.6 referred to above.

The user makes a motion to display an icon image at a desired positionon the image 100 in the state in FIG. 6. The motion is detected by thesensor 13. The PC 10 a generates motion coordinates on the basis of thedetection result of the sensor 13, and displays, at the position of thegenerated motion coordinates, an icon image corresponding to a region inwhich the motion coordinates are included.

When a user object is present near the motion coordinates, the PC 10 acontrols the display of the user object such that the user object movestoward the motion coordinates in accordance with the display of the iconimage. FIG. 8 illustrates an example where the user makes a motion todisplay an icon image at a position to the left from the center in theimage 100 in the state in FIG. 6. It is understood from the example inFIG. 8 that an icon image 130 ₃ is displayed at a position to the leftfrom the center in the image 100, and the user objects 120 ₁, 120 ₄, and120 ₁₃ displayed near the position of the icon image 130 ₃ in the statein FIG. 6 have moved toward the position of the icon image 130 ₃.

The change in state of a user object in response to a motion is notlimited to the movement of the display position of the user object. Forexample, when motion coordinates are included in an image range of auser object, the PC 10 a can display the user object with emphasis. Forexample, the image range of the user object can be defined by the ycoordinates of the upper and lower ends of the user object and the xcoordinates of the right and left ends of the user object. For example,the PC 10 a determines whether the motion contacts with the user objecton the basis of the motion coordinates, and when determining that themotion has contacted with the user object, displays the user object withemphasis. Examples of the emphasized display of the user object includedisplaying the user object on an enlarged scale and vibrating the userobject.

Another example of the emphasized display is that, as exemplified inFIG. 9, when the image range of the user object 120 ₁ includes themotion coordinates, for example, the PC 10 a displays an icon image 130₄ at the position of the motion coordinates, and displays an effectimage 132 ₁ near the icon image 130 ₄. In the example in FIG. 9, theeffect image 132 ₁ uses an image representing that star images arescattered around the icon image 130 ₄. For example, the effect image 132₁ is stored in the DB 12 a in advance.

The PC 10 a can store a screenshot of the image 100 in response to themotion of the user. For example, the PC 10 a determines whether motioncoordinates corresponding to the motion of the user are included in animage range of a screenshot icon image 112 displayed at the upper leftcorner position of the image 100. When the PC 10 a determines that themotion coordinates are included in the image range of the icon image112, as exemplified in FIG. 10, the PC 10 a displays an icon image 131indicating a screenshot at the position of the motion coordinates. ThePC 10 a stores the image 100, in which each user object and icon imagesat the time at which the motion coordinates are determined to beincluded in the image range of the icon image 112 are reflected, in amemory as a screenshot image.

As described above, the PC 10 a holds information on the image 100 to beprojected to the projection medium 14 as coordinate information on the zaxis in the depth direction. Specifically, the image 100 is an imageobtained by projecting information on a three-dimensional spacerepresented by the x axis, the y axis, and the z axis onto atwo-dimensional space (plane) represented by the x axis and the y axis.

Referring to FIG. 11A and FIG. 11B, an example of the configuration ofthe image 100 in the PC 10 a is described. For example, an image 100 isdisplayed on the projection medium 14 as illustrated in FIG. 11A. In theexample in FIG. 11A, the image 100 includes a background image 110including a sky region 110 a and a land region 110 b, a fixed object111, user objects 120 ₂₀, 120 ₂₁, 120 ₂₂, 120 ₂₃, 120 ₂₄, 120 ₂₅, 120₂₆, 120 ₂₇, . . . , an icon image 130 ₅, and an effect image 132 ₂.

The image 100 illustrated in FIG. 11A is displayed by using only the xaxis and the y axis among the x axis, the y axis, and the z axis, andthe z axis is ignored.

In FIG. 11A, for example, the user objects 120 ₂₀, 120 ₂₁, and 120 ₂₂included in the land region 110 b are displayed such that the userobject 120 ₂₀ is displayed foremost and the user object 120 ₂₂ isdisplayed rearmost depending on the degrees of overlapping. Similarly,the user objects 120 ₂₃ and 120 ₂₄ included in the land region 110 b aredisplayed such that the user object 120 ₂₃ is displayed on the frontside and the user object 120 ₂₄ is displayed on the back side.

For a first set of the user objects 120 ₂₀ or 120 ₂₁, and 120 ₂₂ and asecond set of the user objects 120 ₂₃ and 120 ₂₄, the second set is seenso as to be displayed behind the first set on the basis of the positionsin the land region 110 b.

The user object 120 ₂₆ and user object 120 ₂₇ included in the sky region110 a and the user object 120 ₂₅ included in the land region 110 b haveno portions overlapping with each other in the state in FIG. 11A, andhence it is difficult to determine the perspective from the display ofthe image 100.

FIG. 11B is an overhead view of a three-dimensional space obtained byadding z-axis information to the image 100 in FIG. 11A from a virtualviewpoint at the upper right on the front side of the image 100. In FIG.11B, the vertical direction corresponds to the y axis, the directionfrom the upper left toward the lower right corresponds to the x axis,and the direction from the left to the right slightly upward rightcorresponds to the z axis. The sky region 110 a is displayed as an xyplane in which the coordinate z is 0. The land region 110 b is displayedas an xz plane in which the coordinate y is 0. In FIG. 11B, the left endof the line of intersection between the sky region 110 a and the landregion 110 b has a coordinate x of 0.

The image 100 in FIG. 11A corresponds to an image obtained by viewingthe three-dimensional space from the line of sight fixed in thedirection along the z axis as indicated by the arrow A in FIG. 11B. ThePC 10 a determines in advance a region that can be displayed in thethree-dimensional space represented by the x axis, the y axis, and the zaxis. A region determined as the region that can be represented isreferred to as “defined region”.

Each of the user objects 120 ₂₀ to 120 ₂₅ included in the land region110 b is displayed such that, for example, the value of the coordinate zat the lower end of the image range is fixed to 0. Each of the userobjects 120 ₂₆ and 120 ₂₇ included in the sky region 110 a can bedisplayed such that the value of each of the coordinates x, y, and z isfreely set within a defined region.

In FIG. 11B, the coordinate z becomes larger from the left to right inthe figure. As illustrated in FIG. 11B, in the user objects 120 ₂₀, 120₂₁, and 120 ₂₂, the value of the coordinate z is the smallest for theuser object 120 ₂₀ and the largest for the user object 120 ₂₂. Thus,when the user objects 120 ₂₀, 120 ₂₁, and 120 ₂₂ are partiallysuperimposed on one another on the xy plane, the user objects aresuperimposed and displayed on the image 100 in the order of the userobject 120 ₂₂, the user object 120 ₂₁, and the user object 120 ₂₀.

For a set of the user object 120 ₂₃ and 120 ₂₄ (second set), the valuesof the coordinate z are larger than those of a set of the user objects120 ₂₀ to 120 ₂₂ (first set), and the user object 120 ₂₃ and 120 ₂₄ aredisplayed on the deeper side on the xz plane. Thus, when thethree-dimensional space is viewed in the direction indicated by thearrow A with respect to the xz plane in the defined region, the userobject 120 ₂₃ and 120 ₂₄ are displayed on the upper side (the sidecloser to a boundary between the sky region 110 a and the land region110 b) in the image 100.

For the user objects 120 ₂₅ to 120 ₂₇, the user object 120 ₂₆ has thesmallest value of the coordinate z and is displayed on the front side,and the user object 120 ₂₇ has the largest value of the coordinate z andis displayed on the back side. For user objects of the same size, a userobject having a smaller value of the coordinate z is displayed on alarger scale (for example, user objects 120 ₂₃ and 120 ₂₄).

Each of the user objects 120 ₂₀ to 120 ₂₅ included in the land region110 b can freely move on the xz plane within a defined region. Each ofthe user objects 120 ₂₆ and 120 ₂₇ included in the sky region 110 a canfreely move on the xyz plane within a defined region. In this case, thevalue of the coordinate y of each of the user objects 120 ₂₅ and 120 ₂₇included in the sky region 110 a can be limited to a given value ormore.

Configuration that can be Applied in First Embodiment

FIG. 12 illustrates an exemplary configuration of the PC 10 a that canbe applied in the first embodiment. In the PC 10 a in FIG. 12, a centralprocessing unit (CPU) 1001, a read only memory (ROM) 1002, a random,access memory (RAM) 1003, and a display signal generation unit 1004 areconnected to a bus 1000. In the PC 10 a, a storage 1006, a data I/F1007, and a communication I/F 1008 are further connected to the bus1000.

The CPU 1001 uses the RAM 1003 as a work memory to control the overallPC 10 in accordance with computer programs stored in the ROM 1002 andthe storage 1006 in advance. The display signal generation unit 1004 isconnected to a monitor 1005, and converts a display control signalgenerated by the CPU 1001 into a signal that can be displayed by themonitor 1005, and outputs the resultant. The display signal generationunit 1004 can convert the display control signal into a signal that canbe displayed by the projector (PJ) 11, and output the resultant.

The storage 1006 is a storage medium capable of storing therein data ina non-volatile manner, and, for example, a hard disk drive is used.Without being limited thereto, a non-volatile semiconductor memory suchas a flash memory may be used as the storage 1006. The storage 1006stores therein computer programs to be executed by the CPU 1001 andvarious kinds of data.

The data I/F 1007 controls input and output of data to and from anexternal device. For example, the data I/F 1007 is used as an interfacefor the image acquisition device 20. The data I/F 1007 inputs signalsfrom a pointing device such as a mouse and a keyboard (KBD) (not shown),The display control signal generated by the CPU 1001 may be output fromthe data I/F 1007, and supplied to the projector 11, for example. Forthe data I/F 1007, an interface such as a universal serial bus (USB) andBluetooth (trademark) can be applied.

The communication I/F 1008 controls communication via a network such asthe Internet and a local area network (LAN).

FIG. 13 is an exemplary function block diagram illustrating the functionof the PC 10 a that can be applied in the first embodiment. In FIG. 13,the PC 10 a includes an image analysis unit 200 a, a registration unit201, a display control unit 210 a, and a motion analysis unit 220. Theimage analysis unit 200 a, the registration unit 201, the displaycontrol unit 210 a, and the motion analysis unit 220 are implemented bycomputer programs running on the CPU 1001. Without being limitedthereto, a part or whole of the image analysis unit 200 a, theregistration unit 201, the display control unit 210 a, and the motionanalysis unit 220 may be formed by independent pieces of hardware.

A user image DB 230 and a display image DB 231 are connected to the PC10 a. The user image DB 230 and the display image DB 231 are included inthe above-mentioned DB 12 a. The user image DB 230 and the display imageDB 231 are configured by using a storage medium externally connected tothe PC 10 a or a predetermined region in the storage 1006 in the PC 10a.

The image analysis unit 200 a is supplied with images acquired by theimage acquisition device 20. The image analysis unit 200 a analyzes animage supplied from the image acquisition device 20 to acquireidentification information, a user image, and information representingattributes from the image. The registration unit 201 stores theidentification information, the user image, and the attributeinformation acquired by the image analysis unit 200 a in the user imageDB 230 in association with one another. For example, the user image DB230 can store therein icon images and effect images in addition to userimages.

The motion analysis unit 220 is supplied with a detection result outputfrom the sensor 13. The motion analysis unit 220 outputs motioninformation on the basis of the supplied detection result. For example,the motion information includes coordinate information based on thedetection result of the sensor 13. The motion analysis unit 220 mayfurther include time information indicating time corresponding to thetiming at which coordinate information is detected in the motioninformation.

The display control unit 210 a is supplied with an identification image41 from the identification image reading device 21. The display controlunit 210 a acquires identification information from the suppliedidentification image 41. The display control unit 210 a acquires theuser image from the user image DB 230, combines the acquired user imagewith a background image 110 to generate an image 100, and outputs theimage 100. For example, the image 100 output from the display controlunit 210 a is supplied to the projector (PJ) 11 and projected on theprojection medium 14.

In this case, the display control unit 210 a successively acquires userimages that are stored in the user image DB 230 by the registration unit201. The display control unit 210 a acquires, from the user image DB230, a user image associated with identification information based on anidentification image 41 supplied from the identification image readingdevice 21. The display control unit 210 a stores the acquired user imagein the display image DB 231 in association with the identificationinformation. The display control unit 210 a combines each user imagestored in the display image DB 231 with the background image 110, andoutputs the combined image.

The display control unit 210 a can further combine the image 100 with apredetermined icon image or effect image on the basis of the motioninformation supplied from the motion analysis unit 220. The displaycontrol unit 210 a can change each user image included in the image 100on the basis of the motion information supplied from the motion analysisunit 220.

The image analysis unit 200 a, the registration unit 201, the displaycontrol unit 210 a, and the motion analysis unit 220 included in the PC10 a described above is implemented by, for example a computer programstored in the storage 1006 and running on the CPU 1001. The computerprogram is recorded in a computer-readable recording medium, such as acompact disc (CD), a flexible disk (FD), and a digital versatile disc(DVD), as a file in an installable format or an executable format, andprovided.

A computer program to be executed by the PC 10 a in the first embodimentmay be stored on a computer connected to a network such as the Internet,and provided by being downloaded via the network. A computer program tobe executed by the PC 10 a in the first embodiment may be provided ordistributed via a network such as the Internet. A computer program inthe first embodiment may be provided by being incorporated in the ROM1002 in advance.

A computer program executed by the PC 10 a in the first embodiment has amodule configuration including each of the above-mentioned units (imageanalysis unit 200 a, registration unit 201, display control unit 210 a,and motion analysis unit 220). Actual hardware is configured such thatthe CPU 1001 reads a computer program from a storage medium such as thestorage 1006 and the ROM 1002 and executes the read computer program,and each of the above-mentioned units is loaded on a main storage devicesuch as the RAM 1003 so that the image analysis unit 200 a, theregistration unit 201, the display control unit 210 a, and the motionanalysis unit 220 are generated on the main storage device.

FIG. 14 is an exemplary function block diagram illustrating the functionof the image analysis unit 200 a according to the first embodiment. InFIG. 14, the image analysis unit 200 a includes an image acquisitionunit 2001, a user image extraction unit 2002, an identificationinformation acquisition unit 2003, and an attribute informationacquisition unit 2004.

The image acquisition unit 2001 acquires an image supplied from theimage acquisition device 20. The user image extraction unit 2002 detectsa hand-drawing region 43 from the image acquired by the imageacquisition unit 2001, and extracts a user image from an image in thedetected hand-drawing region 43. The identification informationacquisition unit 2003 extracts an identification image 41 from the imageacquired by the image acquisition unit 2001, and decodes the extractedidentification image 41 to acquire identification information. Theattribute information acquisition unit 2004 detects an attribute settingregion 42 from the image acquired by the image acquisition unit 2001,and acquires attribute information from the detected attribute settingregion 42.

FIG. 15 is an exemplary function block diagram illustrating the functionof the display control unit 210 a according to the first embodiment. InFIG. 15, the display control unit 210 a includes an image managementunit 2101, an identification information acquisition unit 2102, an imagecontrol unit 2103 a, a motion information acquisition unit 2104, a 3Dspace generation unit 2105, a region setting unit 2106, a combining unit2107, and a display image acquisition unit 2120.

The identification information acquisition unit 2102 decodes anidentification image 41 supplied from the identification image readingdevice 21 to acquire identification information. The image managementunit 2101 acquires a user image and attribute information associatedwith the identification information from the user image DB 230, suppliesthe identification information, the user image, and the attributeinformation to the image control unit 2103 a, and stores theidentification information, the user image, and the attributeinformation in the display image DB 231. In response to the storage ofthe user image in the user image DB 230 by the registration unit 201,the image management unit 2101 acquires the user image andidentification information and attribute information associated with theuser image from the user image DB 230, and supplies the acquired userimage, identification information, and attribute information to theimage control unit 2103 a.

The motion information acquisition unit 2104 acquires motion informationoutput from the motion analysis unit 220, and supplies the acquiredmotion information to the image control unit 2103 a.

The image control unit 2103 a is supplied with the user image, theattribute information, and the identification information acquired bythe image management unit 2101. For example, the image control unit 2103a generates, on the basis of the user image and the attributeinformation, parameters for controlling the display of the user image,and applies the generated parameters to the user image. The user imageapplied with the parameters is supplied to the combining unit 2107.

For example, when the image control unit 2103 a is supplied with themotion information from the motion information acquisition unit 2104,the image control unit 2103 a changes parameters applied to the userimage in response to the motion information. When the image control unit2103 a is supplied with the motion information from the motioninformation acquisition unit 2104, for example, the image control unit2103 a acquires an icon image from the user image DB 230, and suppliesthe acquired icon image to the combining unit 2107 in association withcoordinate information included in the motion information. The 3D spacegeneration unit 2105 generates a three-dimensional image data spacerepresented by the x axis, the y axis, and the z axis described abovewith reference to FIG. 11A and FIG. 11B. For example, the 3D spacegeneration unit 2105 generates a three-dimensional image data space asan address space on the RAM 1003. The region setting unit 2106 sets adefined region for the three-dimensional image data space in accordancewith values determined in advance for the x axis, the y axis, and the zaxis. An image in the defined region is displayed as the image 100. Auser image and icon images are displayed in the image 100 whencoordinates thereof are included in the defined region. The regionsetting unit 2106 supplies information indicating the defined region tothe combining unit 2107.

On the basis of the defined region, the combining unit 2107 combines thebackground image 110 and the fixed object 111 prepared in advance andthe user image and the icon image supplied from the image control unit2103 a with a two-dimensional image obtained by viewing the definedregion from the line of sight as the direction along the z axis, andoutputs the combined image as the image 100 (see FIG. 11A and FIG. 11B).

The background image 110 and the fixed object 111 can be stored in apredetermined region in the DB 12 a in advance. The background image 110and the fixed object 111 may be stored in a predetermined region in thestorage 1006.

Under control of the image control unit 2103 a, the display imageacquisition unit 2120 acquires the image 100 output from the combiningunit 2107. For example, the image control unit 2103 a instructs thedisplay image acquisition unit 2120 to acquire the image 100 inaccordance with predetermined motion information. The display imageacquisition unit 2120 acquires the image 100 in response to theinstruction, and stores the acquired image 100 in the storage 1006, forexample.

FIG. 16 is an exemplary flowchart schematically illustrating processingfor displaying an image acquired by the image acquisition device 20according to the first embodiment. Prior to the processing in theflowchart in FIG. 16, a user prepares a medium 40 on which anidentification image 41 is displayed and a sheet 50. The user draws apicture 45 in a hand-drawing region 43 on the sheet 50, checks desiredattributes in an attribute setting region 42, sets the sheet 50 on theimage acquisition device 20, places the medium 40 in a placement region44, and causes the image acquisition device 20 to start imageacquisition processing. The image acquisition device 20 transmits theacquired image to the PC 10 a.

The image transmitted from the image acquisition device 20 to the PC 10a is received by the PC 10 a and supplied to the image analysis unit 200a. The image analysis unit 200 a acquires the supplied image (Step S10).At the next Step S11, the image analysis unit 200 a detects thehand-drawing region 43 and the attribute setting region 42 from thesupplied image, and extracts a user image and attribute information fromthe detected regions. The image analysis unit 200 a extracts anidentification image from the supplied image, and decodes the extractedidentification image to acquire identification information.

At the next Step S12, the registration unit 201 stores the user imageand the attribute information extracted at Step S11 in the user image DB230 in association with the identification information, and registersthe user image. The processing proceeds to Step S13. At Step S13, thedisplay control unit 210 a sets parameters p for controlling display ofthe user image.

Now, the parameters p that can be applied in the first embodiment aredescribed. The display control unit 210 a determines, for example, thefollowing eight kinds of parameters p₀ to p₇ for each user image, thatis, for each user object.

(1) p₀: maximum speed v_(max) in travel direction

(2) p₁: acceleration a in travel direction

(3) p₂: maximum value α_(hmax) of angular acceleration in horizontaldirection

(4) p₃: maximum value α_(vmax) of angular acceleration in verticaldirection

(5) p₄: maximum value dR_(umax) of random number width (maximum randomnumber width) in upward angle

(6) p₅: maximum random number width dR_(dmax) in downward angle

(7) p₆: maximum, random number width dR_(rmax) in rightward

(8) p₇: maximum random number width dR_(lmax) in leftward angle

Of those, the maximum speed v_(max) in the travel direction and theacceleration a in the travel direction as the parameters p₀ and p₁ areparameters for controlling the speed of a user object in a defined spacein the travel direction. The maximum value α_(hmax) of the angularacceleration in the horizontal direction and the maximum value α_(vmax)of the angular acceleration in the vertical direction as the parametersp₂ and p₃ are parameters for rotating the user object in the horizontaldirection and the vertical direction.

The maximum random number widths dR_(umax) and dR_(dmax) in the upwardand downward directions as the parameters p₄ and p₅ are parameters forgiving the ranges of angular acceleration with, respect to the rotationoperation of the user object in the vertical direction, that is, in theupward and downward directions with respect to the travel direction. Theangular accelerations given by the parameters p₄ and p₅ are limited bythe maximum value α_(vmax) of the angular acceleration in the verticaldirection as the parameter p₃.

The maximum random number widths dR_(rmax) and dR_(lmax) in the leftwarddirection and the rightward direction as the parameters p₆ and p₇ areparameters for giving the ranges of angular acceleration with respect tothe rotation operation of the user object in the horizontal direction,that is, in the leftward and rightward directions with respect to thetravel direction.

For example, the display control unit 210 a generates, on the basis of auser image and attribute information, the parameters p₀ to p₇ fordetermining the performance related to the motion of a user object forthe user image. For example, the display control unit 210 a determinesthe values of the parameters p₀ to p₇ in accordance with the attributeinformation corresponding to the user image, and further sets the rangewhere each of the determined parameters p₀ to p₇ can be changed on thebasis of the color used for the user image and the shape and size of theuser image.

At the next Step S14, the display control unit 210 a sets coordinates ina defined region for the user image for which the parameters p have beenset, and combines the user image with a background image 110 to generatean image 100. In this manner, the user image is displayed as a userobject in a 3D space defined as the defined region. The coordinates ofthe initial position of the user object may be fixed or may bedetermined randomly on the basis of attribute information.

FIG. 17 is an exemplary flowchart schematically illustrating processingperformed when the identification image 41 is acquired by theidentification image reading device 21 according to the firstembodiment. Prior to the processing in the flowchart in FIG. 17, a userregisters a user image in the user image DB 230 in advance inassociation with identification information in accordance with theabove-mentioned flowchart in FIG. 16, for example.

The user sets a medium 40 on which an identification image 41 isdisplayed on the identification image reading device 21, and instructsthe identification image reading device 21 to read the identificationimage 41 in response to user's operation. In response to thisinstruction, the identification image reading device 21 reads theidentification image 41, and transmits the read identification image 41to the PC 10 a. The identification image reading device 21 may beconfigured to automatically recognize and read the identification image41.

The identification image 41 transmitted from the identification imagereading device 21 to the PC 10 a is received by the PC 10 a and suppliedto the display control unit 210 a. The display control unit 210 aacquires the supplied identification image 41 (Step S20). At the nextStep S21, the display control unit 210 a decodes the identificationimage 41 to acquire identification information, and searches the userimage DB 230 for a user image associated with the acquiredidentification information.

At the next Step S22, the display control unit 210 a determines, on thebasis of each user image retrieved from the user image DB 230,parameters p₀ to p₇ similarly to the above-mentioned processing at StepS13, and sets the parameters p₀ to p₇ for each user image.

At the next Step S23, the display control unit 210 a sets coordinates ina defined region for the user image for which the parameters p have beenset, and combines the user image with a background image 110 to generatean image 100. In this manner, each user image associated withidentification information based on the identification image 41 in theuser image DB 230 is displayed as a user object in a 3D space defined asa defined region.

Referring to FIG. 18 and FIG. 19, the processing for acquiring a userimage and each piece of information from the sheet 50 is described inmore detail. FIG. 18 is an exemplary flowchart illustrating theoperation of the image analysis unit 200 a according to the firstembodiment. FIG. 19 is a diagram for describing processing forextracting each image from the sheet 50 according to the firstembodiment. In FIG. 19, portions common to FIG. 2 and FIG. 4 referred toabove are denoted by the same reference symbols, and detaileddescriptions are omitted.

In the flowchart in FIG. 18, at Step S40, the image analysis unit 200 aacquires an image of the sheet 50, which is supplied from the imageacquisition device 20, from the image acquisition unit 2001. The imageacquisition unit 2001 supplies the acquired image to the user imageextraction unit 2002, the identification information acquisition unit2003, and the attribute information acquisition unit 2004. In this case,the image acquisition unit 2001 recognizes markers 51 ₁ to 51 ₃ (seeleft diagram in FIG. 19) arranged at three corners of the sheet 50,thereby being capable of correcting the orientation of the acquiredimage of the sheet 50.

At Step S41, the identification information acquisition unit 2003determines whether an identification image 41 is included in the imageof the sheet 50. When the identification information acquisition unit2003 determines that the identification image 41 is included in theimage (“present” at Step S41), the identification informationacquisition unit 2003 advances the processing to Step S42 to extract theidentification image 41 from the image. At the next Step S43, theidentification information acquisition unit 2003 decodes the extractedidentification image 41 to acquire identification information. Theprocessing proceeds to Step S45.

On the other hand, when the identification information acquisition unit2003 determines at Step S41 that the identification image 41 is notincluded (“absent” at Step S41), the identification informationacquisition unit 2003 advances the processing to Step S44. At Step S44,for example, the identification information acquisition unit 2003 setspredetermined default identification information as acquiredidentification information. The processing proceeds to Step S45.

At Step S45, the attribute information acquisition unit 2004 extracts animage in the attribute setting region 42 from an image of the sheet 50,and determines whether attributes are designated on the basis of theextracted image of the attribute setting region 42.

For example, as illustrated in the left diagram in FIG. 19, sixattribute setting parts 42 ₁ to 42 ₆ are arranged in the attributesetting region 42. The attribute setting parts 42 ₁ to 42 ₆ are used todesignate desired attributes when checked by filling the inside thereof.In the example in FIG. 19, the attribute setting parts 42 ₁ to 42 ₆ areused to set an attribute “walk”, an attribute “go forward”, an attribute“fly”, an attribute “plant”, an attribute “building”, and an attribute“swim”, respectively. The attributes set in the attribute setting parts42 ₁ to 42 ₆ are referred to as “attribute Type#1”, “attribute Type#2”,“attribute Type#3”, “attribute Type#4”, “attribute Type#5”, and“attribute Type#6”, respectively.

The attribute information acquisition unit 2004 confirms whether theattribute setting parts 42 ₁ to 42 ₆ are checked one by one in apredetermined order. When it is confirmed that any of the attributesetting parts is checked (“present” at Step S45). The attributeinformation acquisition unit 2004 acquires attribute informationindicating the attribute corresponding to the checked attribute settingpart (Step S46). In the example in FIG. 19, the attribute setting part422 is checked, and attribute information indicating the attributeType#2 is acquired. When the attribute information is acquired, theprocessing proceeds to Step S48.

When the attribute information acquisition unit 2004 confirms that noneof the attribute setting parts 42 ₁ to 42 ₆ is checked (“absent” at StepS45), the processing proceeds to Step S47, and the attribute informationacquisition unit 2004 sets the attribute information to a default valuedetermined in advance, for example. The processing proceeds to Step S48.

At the time at which it is confirmed that any of the attribute settingparts 42 ₁ to 42 ₆ has been checked for the first time, the attributeinformation acquisition unit 2004 suspends the confirmation on thepresence/absence of checks in the attribute setting parts. Thus, evenwhen more than one of the attribute setting parts 42 ₁ to 42 ₆ have beenchecked, only one piece of attribute information is acquired.

At Step S48, the user image extraction unit 2002 detects thehand-drawing region 43 from the image of the sheet 50, and extracts auser image in the hand-drawing region 43. The user image extraction unit2002 performs user object extraction processing on the image in thehand-drawing region 43. For example, the user image extraction unit 2002performs a binary determination on the image in the hand-drawing region43 as to whether each pixel is white (basic color of sheet 50) or colorother than white, and extracts the portion of the picture 45 as a userobject. The user image extraction unit 2002 sets an image in a minimumrectangular region 46 that includes the extracted portion of the picture45 and has a base direction parallel to the base direction of thehand-drawing region 43 as a user image.

After the user image is extracted at Step S48, the image analysis unit200 a requests the registration unit 201 to register the identificationinformation, the attribute information, and the user image acquired orextracted by the processing at Steps S40 to S48 in the user image DB 230(Step S49).

In the above description, the processing of the user image extractionunit 2002, the processing of the identification information acquisitionunit 2003, and the processing of the attribute information acquisitionunit 2004 are executed in series, but the execution method is notlimited to this example. Specifically, the user image extraction unit2002, the identification information acquisition unit 2003, and theattribute information acquisition unit 2004 may be executed in parallel.

FIG. 20 is an exemplary flowchart illustrating processing forregistering identification information, attribute information, and auser image in the user image DB 230 by the registration unit 201according to the first embodiment. At Step S60, the registration unit201 determines whether an instruction to register identificationinformation, attribute information, and a user image in the user imageDB 230 has been issued from the image analysis unit 200 a. When theregistration unit 201 determines that there is no registeringinstruction (“absent” at Step S60), the registration unit 201 returnsthe processing to Step S60.

On the other hand, when the registration unit 201 determines that aninstruction to register identification information, attributeinformation, and a user image in the user image DB 230 has been issued(“present” at Step S60), the processing proceeds to Step S61. At StepS61, the registration unit 201 receives the identification information,the attribute information, and the user image requested to be registeredfrom the image analysis unit 200 a.

At the next Step S62, the registration unit 201 generates managementinformation for managing the identification information, the attributeinformation, and the user image received at Step S61. For example, theregistration unit 201 generates the management information on the basisof the identification information. As exemplified in FIG. 21, theregistration unit 201 generates the management information such that theattribute information and the user image are associated with theidentification information. The user image and the attribute informationare associated with each other on a one-to-one basis. On the other hand,the identification information has a one-to-many relation with a set ofthe user image and the attribute information. Specifically, each pieceof identification information can be associated with sets of user imagesand attribute information.

At the next Step S63, the registration unit 201 generates a managementregion for storing the identification information, the attributeinformation, and the user image received at Step S61 in the user imageDB 230. The registration unit 201 generates the management region on thebasis of the above-mentioned management information generated at StepS62.

In this example, the registration unit 201 generates the managementregion by using a hierarchical structure in a file system of a computer.Referring to FIG. 22, an example of generating a management region byusing a folder structure in a file system, which is generated by theregistration unit 201, is described. FIG. 22 illustrates an example of adisplay screen indicating a folder structure of the user image DB 230according to the first embodiment. In FIG. 22, a display screen 300includes display regions 301 and 302. The display region 301 indicatesthe folder structure, and the display region 302 indicates the contentsof a folder designated in the display region 301.

In the folder structure, a folder on a certain level can include afolder on the next level below the certain level, and the hierarchies offolders are sequentially formed from a folder on an upper level to afolder on a lower level. In the display region 301 in FIG. 22, a firstlevel folder 320 (folder “DataBase”) includes second level folders 321₁, 321 ₂, 321 ₃, . . . , 321 _(m), 321 _(m+1), 321 _(m+2), . . . (firstregion). Each of the second level folders 321 ₁, 321 ₂, . . .corresponds to identification information, and a character string of theidentification information is used as the folder name.

The second level folder further includes a third level folder (secondregion). For example, in FIG. 22, a second level folder 321 _(m)includes third level folders 322 ₁ to 322 ₅. A set of a user image andattribute information is stored in each of the third level folders 322 ₁to 322 ₅. For the third level folder, time information indicating timeat which a user image and attribute information stored in the folderwere acquired by the image analysis unit 200 a is used as the foldername.

In this manner, the folder name of the third level folder is determinedon the basis of the time information, and hence the uniqueness of a setof user images and attribute information associated with the sameidentification information is guaranteed. The folder name of the secondlevel folder is determined with use of identification information, andhence a set of a user image and attribute information associated withthe same identification information as those in a second level folder isadded and stored in the folder.

Such a configuration of the management region enables identificationinformation, user images, and attribute information to be managed withuse of a file system of an operating system (OS) installed on the PC 10a, which makes it easy to acquire a set of user images and attributeinformation associated with a designated identification image.

In FIG. 22, the contents of the third level folder 322 ₁ are indicatedin the display region 302. In this example, the display region 302indicates that a file 310 for storing therein a user image and a file311 for storing therein attribute information are stored in the thirdlevel folder 322 ₁.

FIG. 23 illustrates an example of the configuration of the file 311 inwhich the attribute information is stored according to the firstembodiment. In this example, the file 311 has stored therein attributeinformation 312 and identification information 313 associated with theattribute information 312.

Referring back to FIG. 20, after the registration unit 201 generates themanagement region in the user image DB 230 at Step S63 as describedabove, at the next Step S64, the registration unit 201 stores the userimage, the attribute information, and the identification informationreceived from the image analysis unit 200 a at Step S61 in the generatedmanagement region.

The user image, the attribute information, and the identificationinformation stored in the management region at Step S64 are read to thedisplay control unit 210 a for display. The display control unit 210 areads the newly registered user image, attribute information, andidentification information from the user image DB 230. The displaycontrol unit 210 a generates a display queue region in the display imageDB 231 (Step S65). For the display queue region, the same configurationas that of the management region generated at Step S63 can be applied,and hence the description of the configuration is omitted.

The display control unit 210 a stores the user image, the attributeinformation, and the identification information, which have been readfrom the user image DB 230, in the generated display queue region (StepS66). The display control unit 210 a uses the user image stored in thedisplay queue region to display a user object in the image 100.

At the next Step S67, the display control unit 210 a counts the numberof user images stored in the display queue region. For example, thedisplay control unit 210 a counts a value obtained by totaling thenumbers of third level folders in each of which a set of a user imageand attribute information is stored for each second level foldercorresponding to identification information in the display queue regionas the number of user images stored in the display queue region. At thenext Step S68, the display control unit 210 a determines whether thenumber of user images counted at Step S67 has exceeded a predeterminednumber. When the display control unit 210 a determines that the countednumber of user images has not exceeded the predetermined number (“No” atStep S68), the processing returns to Step S60.

On the other hand, the display control unit 210 a determines that thecounted number of user images has exceeded the predetermined number(“Yes” at Step S68), the processing proceeds to Step S69, and thedisplay control unit 210 a deletes one user image in the display queueregion. For example, the display control unit 210 a deletes a foldergenerated at the oldest time among third level folders in the displayqueue region. In this manner, the number of user objects displayed inthe image 100 can be limited to a constant number. The number of userimages deleted at Step S69 is not limited to one, and two or more userimages may be deleted.

The deletion of a user image designated from among user images stored inthe display queue region can be inhibited. For example, in the displayqueue region, a second level folder with a predetermined folder name iscreated, and a user image to be inhibited from being deleted andattribute information corresponding to the user image are stored in athird level folder created in the second level folder. For example, alogo mark image of a service provider is stored in the folder as a userimage. Even when user images are sequentially stored in the displayqueue region, the logo mark image is not deleted but continues to bedisplayed in the image 100.

Motion Determination Processing According to First Embodiment

Next, motion determination according to the first embodiment isdescribed. First, the detection of an object by the sensor 13 isschematically described with reference to FIGS. 24A and 24B. FIGS. 24Aand 24B illustrate an example of a detection range of the sensor 13according to the first embodiment. FIG. 24A illustrates examples of thedetection range of the sensor 13 when viewed in a first directionperpendicular to a projection surface of the projection medium 14. FIG.24B illustrates an example of the detection range of the sensor 13 whenviewed in a second direction parallel to the projection surface. In theexamples in FIG. 24A and FIG. 24B, the sensor 13 is installed such thata detection surface thereof faces upward, that is, the detection surfaceis substantially parallel to the lower end of the projection medium 14and perpendicular to the projection surface of the projection medium 14.

Referring to FIG. 24A and FIG. 24B, the sensor 13 is capable ofdetecting an object in the range of an angle θ₁ on a plane perpendicularto the first direction and in the range of an angle θ₃ on a planeperpendicular to the second direction. In the first embodiment, thedetection range of the sensor 13 on the plane perpendicular to the firstdirection is limited to the range of an angle θ₂ smaller than the angleθ₁, and the detection range of the sensor 13 on the plane perpendicularto the second direction is limited to the range of an angle θ₄ smallerthan the angle θ₃.

For example, the angle θ₂ may be set such that the detection range ofthe sensor 13 does not exceed the width of the projection medium 14 atthe upper end of the projection medium 14. It is preferred that theangle θ₄ be sufficiently smaller than the angle θ₃, and, for example, bean angle that allows the detection range as seen from the firstdirection to be regarded as a plane.

In this manner, by setting the detection range of the sensor 13 to benarrower than that of the capacity of the sensor 13, unintended motionsof a user for the display system 1 a are prevented from being detectedby the sensor 13. As illustrated in FIG. 24B, when the detection rangein the direction perpendicular to the projection surface is limited toan extremely small angle θ₄, the coordinates on the plane parallel tothe projection surface can be easily

Position detection of an object by the sensor 13 is described morespecifically with reference to FIG. 24A and FIG. 24B. In one example, acase where the positions of objects 80 a and 80 b are detected isconsidered. For example, the objects 80 a and 80 b are human arms thrownout toward the projection medium 14. Objects to be detected by thesensor 13 are not limited to the human body.

The sensor 13 can detect an object in the detection range, and detectthe distance of the detected object from the sensor 13 and the positionof the object in a plane parallel to the detection surface of the sensor13. As described above, the detection range of the sensor 13 is limitedto extremely narrow in a direction perpendicular to the projectionsurface. Thus, as illustrated in FIG. 24B, the objects 80 a and 80 b aredetected in ranges 81 a and 81 b that are narrowed extremely in thedirection of the projection surface. In this case, the ranges 81 a and81 b can each be regarded as a place parallel to the projection surface,and the orientations and angles of the objects 80 a and 80 b can beignored. Thus, even when the objects 80 a. and 80 b have a length in thedirection, perpendicular to the projection surface, the position of theobject in the plane parallel to the projection surface can be uniquelydetermined.

For example, the sensor 13 determines, for the detected object 80 a, adistance y₁ from the sensor 13 and a position x₁ of the object 80 a in aplane perpendicular to the detection surface of the sensor 13 andparallel to the projection surface. In this example, the position x₁ isdetermined as a distance from the line passing through the center of thesensor 13, but the determination method is not limited to this example.Similarly, the sensor 13 determines a distance y₂ and a position x₂ forthe object 80 b.

The sensor 13 transmits a pair of the distance y₁ and the position x₁ tothe PC 10 a as a detection result. Similarly, the sensor 13 transmits apair of the distance y₂ and the position x₂ to the PC 10 a as adetection result. The PC 10 a receives the detection results, andsupplies the received detection results to the motion analysis unit 220.The motion analysis unit 220 determines coordinates in the image 100 onthe basis of the supplied detection results. FIG. 25 illustrates anexample of coordinates of the objects 80 a and 80 b in the image 100determined in the manner described above. In the example in FIG. 25, thecoordinates of the object 80 a are determined as coordinates (x₁₀, y₁₀),and the coordinates of the object 80 b are determined as coordinates(x₂₀, y₂₀). As described above, the motion analysis unit 220 has thefunction as a coordinate generation unit configured to generatecoordinates in the image 100 on the basis of detection results of thesensor 13.

The motion analysis unit 220 supplies the thus determined coordinates ofthe objects 80 a and 80 b to the display control unit 210 a. The displaycontrol unit 210 a determines the operation (motion) of the objects 80 aand 80 b on the basis of the coordinates supplied from the motionanalysis unit 220 and the time information indicating the time at whichthe coordinates are supplied. The display control unit 210 a controlsthe display of an icon image and the display of each user object withrespect to the image 100 on the basis of the determined motion.

Attributes of User Object According to First Embodiment

Next, attributes set for a user object according to the first embodimentare described.

First, as described above, regions are set in the image 100 in the firstembodiment. FIG. 26 illustrates an example of the settings of theregions in the image 100 according to the first embodiment. In theexample in FIG. 26, a region a, a region b, and a region c indicating asky region, a land region, and a water region, respectively, are set inthe image 100. In this case, the sky region assumes a sky, that is, aspace in the air, the land region assumes a land, and the water regionassumes a lake or a sea. Regions to be set in the image 100 are notlimited to the sky region, the land region, and the water region.Coordinate information indicating the region a, the region b, and theregion c are stored in the storage 1006, for example.

Next, each attribute set for a user object according to the firstembodiment is described in more detail. In the first embodiment,attributes are set for a user image, that is, a user object. FIG. 27illustrates only the attribute setting region 42 arranged on the sheet50 according to the first embodiment. In the example in FIG. 27, sixattribute setting parts 42 ₁ to 42 ₆ are arranged in the attributesetting region 42. An attribute set in the attribute setting region 42is associated with a user object based on a picture 45 drawn in thehand-drawing region 43 on the sheet 50.

Of the six attributes exemplified in FIG. 27, the attribute “walk”(attribute Type#1) is an attribute corresponding to a motion indicatinga manner that a human or an animal walks, and the motion moves at afirst speed (low speed), for example. The attribute “go forward”(attribute Type#2) is an attribute corresponding to a motion indicatinga manner that a car moves, and the motion moves at a second speed (highspeed) higher than the first speed, for example. The attribute “fly”(attribute Type#3) is an attribute corresponding to a motion indicatinga manner that a bird flies, and the motion moves at a third speed(middle speed) at the middle between the first speed and the secondspeed, for example. The attribute “plant” (attribute Type#4) is anattribute corresponding to a motion indicating a plant, which does notmove. The attribute “building” (attribute Type#5) is an attributeindicating a building, which does not move. The attribute “swim”(attribute Type#6) is an attribute corresponding to a motion indicatinga manner that a fish swims in the water, and the motion moves at theabove-mentioned third speed, for example.

In the first embodiment, each of the attributes is associated with eachregion set in the image 100. FIG. 28 illustrates an example ofassociation between an attribute and a region according to the firstembodiment. In the example in FIG. 28, the attribute “fly” is associatedwith the region a (sky region) in the image 100. Each of the attributes“walk”, “go forward”, “plant”, and “building” is associated with theregion b (land region). The attribute “swim” is associated with theregion c (water region).

The arrangement position of a user object in the image 100 is limited inaccordance with an attribute set for the user object. In the example inFIG. 28, for example, the arrangement of a user object to which any oneof the attributes “walk”, “go forward”, “plant”, and “building” has beenset is limited to the region b as the land region. More specifically, auser object to which the attributes are set is set such that thecoordinate z of a lower end of the image range has a value of 0. Foranother example, the arrangement of a user object to which the attribute“swim” has been set is limited to the region c as the water region. Inthis case, for example, a user object to which the attribute “swim” hasbeen set is controlled such that the value of the coordinate z at thecenter in the image range is 0, thereby indicating that the user objectrelates to water.

In the first embodiment, an attribute and an icon image are furtherassociated with each other. FIG. 29 illustrates an example ofassociation between an attribute and an icon image according to thefirst embodiment. In the example in FIG. 29, icon images Icon#1, Icon#2,Icon#3, Icon#4, Icon#5, and Icon#6 that represent “apple”, “grape”,“rice ball”, “watering pot”, “paint”, and “feed”, respective, aredefined as icon images. The icon image Icon#1 is associated with theattributes Type#1 and Type#2. Similarly, the icon image Icon#2 isassociated with an attribute Type#3, the icon image Icon#4 is associatedwith an attribute Type#4, the icon image Icon#5 is associated with anattribute Type#5, and the icon image Icon#6 is associated with anattribute Type#6. The icon image Icon#3 is associated with all theattributes Type#1 to Type#6.

Each of the icon images Icon#1 to Icon#6 is associated with an effect.In the example in FIG. 29, the icon images Icon#1, Icon#2, and Icon#6are associated with the effect “speed-up” for temporarily increasing themovement speed of the user object, and the icon images Icon#4 and Icon#5are associated with the effect “enlargement” for temporarily enlargingthe display size of the user object. The icon image Icon#3 is associatedwith the effect “speed-up” and the effect “enlargement”.

Each of the icon images Icon#1 to Icon#6 is stored in the user image DB230, for example. Without being limited thereto, each of the icon imagesIcon#1 to Icon#6 may be stored in a predetermined region in the storage1006.

Motion Determination Processing According to First Embodiment

Next, motion determination processing according to the first embodimentis described. FIG. 30 is an exemplary flowchart illustrating motiondetermination processing by the display control unit 210 a according tothe first embodiment. At the first Step S80, the display control unit210 a determines whether an object has been detected in a detectionrange of the sensor 13 on the basis of a detection result of the sensor13. When the display control unit 210 a determines that no object hasbeen detected (“No” at Step S80), the display control unit 210 a returnsthe processing to Step S80. When the display control unit 210 adetermines that an object has been detected (“Yes” at Step S80), on theother hand, the display control unit 210 a advances the processing toStep S81.

At Step S81, the display control unit 210 a acquires, on the basis ofthe detection result of the sensor 13, coordinate information indicatingcoordinates of the position at which the object is detected in the image100. At the next Step S82, the display control unit 210 a determineswhich of regions set in the image 100 the coordinates indicated by thecoordinate information acquired at Step S81 are included.

At the next Step S83, the display control unit 210 a acquires an iconimage. In this case, for example, the display control unit 210 arandomly selects an icon image from among the above-mentioned iconimages Icon#1 to Icon#6. Without being limited thereto, the displaycontrol unit 210 a may select the icon images Icon#1 to Icon#6 inaccordance with a predetermined order.

At the next Step S84, the display control unit 210 a determines whetheranother icon image has already been displayed at the coordinates in theimage 100 indicated by the coordinate information acquired at Step S81.In this case, the display control unit 210 a can make the determinationwith a margin for the coordinates indicated by the coordinateinformation acquired at Step S81. When the display control unit 210 adetermines that another icon image is not displayed (“ABSENT” at StepS84), the display control unit 210 a advances the processing to StepS88, and displays the icon image acquired at Step S83 in the image 100on the basis of the coordinate information acquired at Step S81.

On the other hand, when the display control unit 210 a determines atStep S84 that another icon image has already been displayed at thecoordinates (“PRESENT” at Step S84), the display control unit 210 aadvances the processing to Step S85. At Step S85, the display controlunit 210 a determines whether the displayed icon image is an icon image112 for acquiring a screenshot. When the display control unit 210 adetermines that the icon image is the icon image 112 (“Yes” at StepS85), the display control unit 210 a advances the processing to StepS91. The display control unit 210 a stores the image 100 that reflectseach user object and the icon image at that time point in a memory orthe like as a screenshot image, and acquires a display image by theimage 100. The display control unit 210 a finishes a series ofprocessing in the flowchart in FIG. 30.

When the display control unit 210 a determines at Step S85 that thedisplayed icon image is not the icon image 112 (“No” at Step S85), thedisplay control unit 210 a advances the processing to Step S86. At StepS86, the display control unit 210 a waits for the lapse of apredetermined time. The predetermined time is a relatively short time,such as 1 second or less and several seconds. When the display controlunit 210 a determines that the predetermined time has not elapsed (“No”at Step S86), the display control unit 210 a returns the processing toStep S86. On the other hand, when the display control unit 210 adetermines that the predetermined time has elapsed (“Yes” at Step S86),the display control unit 210 a advances the processing to Step S87.

At Step S87, the display control unit 210 a determines whether theobject detected at Step S80 has been continuously detected. For example,the display control unit 210 a may acquire the coordinates of thecurrently detected object on the basis of the detection result of thesensor 13, determine a difference between the acquired coordinates andthe coordinates acquired at Step S81, and determine that the object hasbeen continuously detected when the determined difference is within apredetermined range. In this manner, the display control unit 210 a candetect the motion of the object detected by the sensor 13, and thedisplay control unit 210 a functions as a motion detection unitconfigured to detect the motion of an object. When the display controlunit 210 a determines that the object has been continuously detected(“Yes” at Step S87), the display control unit 210 a advances theprocessing to Step S89.

At Step S89, the display control unit 210 a moves the icon image thathas been determined to be displayed at Step S84. For example, thedisplay control unit 210 a moves the icon image determined to bedisplayed at Step S84 to the coordinates acquired at Step S87.

On the other hand, when the display control unit 210 a determines atStep S87 that the object detected at Step S80 has not already beendetected (“No” at Step S87), the processing proceeds to Step S90. AtStep S90, the display control unit 210 a deletes the icon imagedetermined to have already been displayed at Step S84, and displays theicon image acquired at Step S83 at the coordinates indicated by thecoordinate information acquired at Step S81. In this manner, the iconimages are switched.

When any one of the above-mentioned processing at Step S88, Step S89,and Step S90, the display control unit 210 a advances the processing toStep S92. At Step S92, the display control unit 210 a determines whetherthe icon image displayed at Step S88, Step S89, or Step S90 contactswith each user object displayed in the image 100.

For example, the display control unit 210 a acquires, on the basis ofthe coordinates and size of the icon image displayed at Step S88, StepS89, or Step S90, the coordinates indicating the range of the icon imagein the image 100. The display control unit 210 a acquires, on the basisof the position and size of each user object that has already beendisplayed in the image 100, coordinates indicating the range of eachuser object in the image 100.

The display control unit 210 a determines, on the basis of the acquiredcoordinates of the ranges of the icon image and the user object, whetherthe range of the icon image overlaps with the range of the user object.When the result of the determination indicates that the range of theicon image overlaps with the range of the user object as a result of thedetermination, the display control unit 210 a determines that the iconimage is in contact with the user object (contact is present). When thedisplay control unit 210 a determines that contact is absent (“contactis absent”at Step S92), the display control unit 210 a advances theprocessing to Step S95.

On the other hand, when the display control unit 210 a determines atStep S92 that contact is present (“contact is present” at Step S92), thedisplay control unit 210 a advances the processing to Step S93. At StepS93, the display control unit 210 a determines whether an attribute setfor the user object determined at Step S92 to be in contact with theicon image corresponds to the attribute of the icon image based on whichthe contact determination is made (icon image displayed at Step S88,Step S89, or Step S90).

For example, the attribute set for the user object that has beendetermined to be in contact with the icon image is the attribute Type#1,and the icon image based on which the determination is made is the iconimage Icon#1. In this case, in the example in FIG. 29, because theattributes associated with the icon image Icon#1 are the attributesType#1 and Type#2, the display control unit 210 a determines that theattribute set for the user object determined at Step S92 to be incontact with the icon image corresponds to the attribute of the iconimage based on which the contact determination is made.

When the display control unit 210 a determines at Step S93 that theattribute set for the user object does not correspond to the attributeof the icon image (“No” at Step S93), the display control unit 210 afinishes a series of processing in the flowchart in FIG. 30. On theother hand, when the display control unit 210 a determines at Step S93that the attribute set for the user object corresponds to the attributeof the icon image (“Yes” at Step S93), the display control unit 210 aadvances the processing to Step S94. At Step S94, for example, thedisplay control unit 210 a acquires an effect image from, the DB 12 a,and displays the acquired effect image (effect) in the image 100 on thebasis of the coordinates of the icon image determined to be in contactwith the user object. After displaying the effect, the display controlunit 210 a advances the processing to Step S95.

At Step S95, for example, when the display control unit 210 a determinesat Step S92 that contact is absent, the display control unit 210 achanges the display of a user object, displayed near the icon image inthe image 100. In this case, the display control unit 210 a selectivelychanges the display of a user object to which an attribute associatedwith the icon image has been set. For example, the display control unit210 a changes the display of the user object such that the user objectmoves toward the icon image at a predetermined speed.

For example, when the display control unit 210 a determines at Step S92that contact is present, the display control unit 210 a changes thedisplay of a user object which is in contact with the icon image and towhich an attribute associated with the icon image has been set. In thiscase, the display control unit 210 a can display the user object withemphasis by swinging the user object, for example.

FIGS. 31A and 31B illustrate an example of the motion of user objects inthe image 100 when it is determined at Step S92 that contact is absentaccording to the first embodiment. For example, as illustrated in FIG.31A, in the image 100, user objects 90 ₁, 90 ₂, 90 ₃, . . . , and 90_(x) to which the attribute Type#3 (fly) has been set originally move inrandom directions in accordance with set corresponding parameters p asindicated by the arrows in FIG. 31A.

In this state, the objects are detected in accordance with theprocessing in the flowchart in FIG. 30, and, for example, an icon image91 as the icon image Icon#3 (rice ball) illustrated in FIG. 29 isdisplayed in accordance with the detection results as illustrated inFIG. 31B. The icon image Icon#3 corresponds to all the attributes Type#1to Type#6. Thus, the display control unit 210 a controls the operationof each of the user objects 90 ₁, 90 ₂, 90 ₃, . . . , and 90 _(x) suchthat the user objects move toward the icon image 91 at predeterminedspeeds as indicated by the arrows in FIG. 31B.

Modification of First Embodiment

Next, a modification of the first embodiment is described. Themodification of the first embodiment relates to display controlperformed when a motion is not detected by the sensor 13 for arelatively long period in addition to the display control in theabove-mentioned first embodiment.

More specifically, in the modification of the first embodiment, when amotion is not detected by the sensor 13 for a relatively long period,such as several minutes or more, in the state in which each user objectin the image 100 moves randomly in accordance with parameters p, eachuser object is caused to execute a predetermined operation. In thismanner, the lines of sight of users who visit an event venue where aservice is provided by the display system can be gathered.

In the modification of the first embodiment, the above-mentioned displaysystem 1 a according to the first embodiment can be directly used, andhence the description of each configuration is omitted.

FIG. 32 is an exemplary flowchart illustrating motion determinationprocessing by the display control unit 210 a according to themodification of the first embodiment. In FIG. 32, processingcorresponding to the flowchart in FIG. 30 referred to above is denotedby the same reference symbol, and a detailed description is omitted.

In FIG. 32, at the first. Step S80′, the display control unit 210 adetermines whether an object has been detected in a detection range ofthe sensor 13 on the basis of a detection result of the sensor 13. Whendetermining that an object has been detected (“Yes” at Step S80′), thedisplay control unit 210 a advances the processing to Step S81. Whendetermining that no object has been detected (“No” at Step S80′), on theother hand, the display control unit 210 a advances the processing toStep S800.

At Step S800, for example, the display control unit 210 a determineswhether a first predetermined period has elapsed since the object wasdetected by the sensor 13. When the display control unit 210 adetermines that the first, predetermined period has not elapsed (“No” atStep S800), the display control unit 210 a returns the processing toStep S80′.

For example, the first predetermined period is a period for whichwhether the lines of sight of users who visit an event venue where aservice is provided by the display system 1 a gather on the image 100projected on the projection medium 14 in the event venue can bedetermined. Examples of the first predetermined period include severalminutes or longer.

Step S86′ in FIG. 32 is common in processing to Step S86 in theflowchart in FIG. 30. A period used to determine the elapse is a secondpredetermined period, which discriminates from the period (firstpredetermined period) determined at Step S800.

When the display control unit 210 a determines at Step S800 that thefirst predetermined period has elapsed (“Yes” at Step S800), the displaycontrol unit 210 a advances the processing to Step S801. At Step S801,the display control unit 210 a controls the display of the image 100such that the operation of each user object displayed in the image 100becomes a predetermined operation.

Examples of the predetermined operation include an operation in whichuser objects that have moved in random directions now move in onedirection. Another example of the predetermined operation is anoperation in which the movement of each user object is stopped and eachuser object swings in a predetermined manner on the moment. Stillanother example of the predetermined operation is an operation in whicheach user object represents an image of a predetermined characterstring.

After the operation of each user object is controlled at Step S801, thedisplay control unit 210 a returns the processing to Step S80′. When thedisplay control unit 210 a determines at Step S80′ after the processingat Step S801 that an object has been detected on the basis of adetection result of the sensor 13, for example, the display control unit210 a may return the operation of each user object to the randomoperation in accordance with the parameters p.

The processing after Step S81 is not different at all from theprocessing after Step S81 in the above-mentioned flowchart in FIG. 30,and hence the descriptions thereof are omitted.

Second Embodiment

Next, a second embodiment is described. In the second embodiment, soundinformation is acquired, and a user object displayed in an image 100 ischanged on the basis of the acquired sound information. FIG. 33illustrates an exemplary configuration of a display system according tothe second embodiment. In FIG. 33, portions common to FIG. 1 referred toabove are denoted by the same reference symbols, and detaileddescriptions are omitted.

In FIG. 33, the illustration of the image acquisition device 20 and theidentification image reading device 21 illustrated in FIG. 1 is omitted.For a PC 10 d illustrated in FIG. 33, the configuration described abovewith reference to FIG. 12 can be directly applied.

In FIG. 33, in a display system 1 d, a microphone 15 for collectingsound is installed near a projection medium 14, and a sound signaloutput from the microphone 15 is supplied to the PC 10 d. The soundsignal output from the microphone 15 in an analog format is supplied tothe PC 10 d after being converted into a digital format sound signal byan A/D converter. For example, the digital format sound signal is inputto a data I/F 1007 included in the PC 10 d.

A DB 12 d stores therein information related to sound in addition to theinformation stored in the above-mentioned DB 12 a. For example, the DB12 d can store therein a sound pattern and a user image in associationwith each other.

With such a configuration, a user stores a user image and attributeinformation in the DB 12 d in association with identificationinformation in advance. The user designates a sound pattern, and storesthe designated sound pattern in the DB 12 d in association with theidentification information in advance. In this state, the user generatessound corresponding to the registered sound pattern, for example, at aposition at which the sound can be collected by the microphone 15. Forexample, the sound pattern is a given rhythm pattern and can beexpressed by handclaps.

The sound corresponding to the sound pattern is collected by themicrophone 15 and converted into a sound signal to be supplied to the PC10 d. The PC 10 d analyzes the supplied sound signal to extract a soundpattern, and searches the DB 12 d for a sound pattern that matches withthe extracted sound pattern. When a sound pattern matching with theextracted sound pattern is retrieved from the DB 12 d as a result, thePC 10 d acquires identification information associated with theretrieved sound pattern.

When user objects for a user image associated with the acquiredidentification information have already been displayed in an image 100,each user object moves toward the position at which the sound wasgenerated, that is, the position of the microphone 15. In this case, forexample, as exemplified in FIG. 34, a user object that is present in apredetermined range 17 in the image 100 with respect to the position 16of the image 100 corresponding to the position of the microphone 15 isselectively controlled to move toward the position 18.

When any user object for a user image associated with the acquiredidentification information is not displayed in the image 100, the PC 10d acquires the user image associated with the identification informationfrom the DB 12 d, and displays the acquired user image in the image 100as a user object. In this case, an appearance position of the userobject can be determined in advance. For example, the appearanceposition may be either a right or left end portion in the image 100.

FIG. 35 is an exemplary function block diagram illustrating the functionof the PC 10 d according to the second embodiment. In FIG. 35, portionscommon to FIG. 13 referred to above are denoted by the same referencesymbols, and detailed descriptions are omitted. In FIG. 35, the PC 10 dis obtained by adding a voice processing unit 240 to the configurationof the PC 10 a illustrated in FIG. 13. An image analysis unit 200 d, aregistration unit 201 d, and a display control unit 210 d are obtainedby adding the function related to processing of sound information to theimage analysis unit 200, the registration unit 201, and the displaycontrol unit 210 a illustrated in FIG. 13. A user image DB 230 can storetherein identification information or a user image in association withsound information as well.

FIG. 36 is an exemplary function block diagram illustrating the functionof the display control unit 210 d according to the second embodiment. InFIG. 36, portions common to FIG. 15 referred to above are denoted by thesame reference symbols, and detailed descriptions are omitted. In FIG.36, the display control unit 210 d is obtained by adding a sound patterndetection unit 2108 to the display control unit 210 a illustrated inFIG. 15. An image management unit 2101 d and an image control unit 2103d are obtained by adding the function for performing processing based onsound information to the image management unit 2101 and the imagecontrol unit 2103 a described above, respectively.

A sound signal output from the microphone 15 and converted into adigital format signal is supplied to the voice processing unit 240. Thevoice processing unit 240 performs predetermined signal processing onthe supplied sound signal, and supplies the resultant to the displaycontrol unit 210 d as sound information. The sound information issupplied to the sound pattern detection unit 2108 in the display controlunit 210 d. The sound pattern detection unit 2108 detects a soundpattern on the basis of the supplied sound information. The detectedsound pattern is supplied to the image management unit 2101 d. The imagemanagement unit 2101 d searches the user image DB 230 on the basis ofthe supplied sound pattern, and supplies the identification informationor the information indicating the user image obtained as a result of thesearch to the image control unit 2103 d.

FIGS. 37A and 37B illustrate an exemplary format used for a user todesignate a sound pattern according to the second embodiment. Asillustrated in FIG. 37A, a pattern designation unit 421 for designatinga sound pattern is provided in a sound pattern designation region 420.In this example, the pattern designation unit 421 is formed by arrangingeight checkboxes. The eight checkboxes can correspond to octuple beats,such that a blank checkbox is regarded as a rest, and a filled andchecked checkbox is regarded as a note. FIG. 37B illustrates an exampleof a pattern designation unit 421′ in which some of the checkboxes arefilled. For example, when sound is generated at a given speed inaccordance with notes and rests indicated by the pattern designationunit 421′, a sound pattern designated by the pattern designation unit421′ can be generated.

For example, the sound pattern designation region 420 exemplified inFIG. 37A is arranged on the sheet 50 in addition to the attributesetting region 42, the hand-drawing region 43, and the placement region44. In this case, a sound pattern designated in the sound patterndesignation region 420 is associated with a user image. Without beinglimited thereto, the sound pattern designation region 420 may bearranged on a sheet different from the sheet 50 on which the attributesetting region 42, the hand-drawing region 43, and the placement region44 are arranged. In this case, for example, a sound pattern designatedin the sound pattern designation region 420 can be associated withidentification information.

In the following, for the sake of description, the sound patterndesignation region 420 is arranged on the sheet 50 together with theattribute setting region 42, the hand-drawing region 43, and theplacement region 44. In other words, a sound pattern is associated witha user image.

Also in this case, for example, a sheet including the sound patterndesignation region 420 is prepared in addition to the sheet 50, and isplaced on the sheet 50 similarly to the medium 40. In this manner, thesound pattern can be associated with user images.

In the above description, the sound pattern is directly designated inthe sound pattern designation region 420, but the designation method isnot limited to this example. For example, a desired sound pattern may beselected from sound patterns prepared in advance.

In the above description, a checkbox in the sound pattern designationregion 420 is checked to register a sound pattern, but the registrationmethod is not limited to this example. For example, a sound pattern maybe registered on the basis of a sound pattern that has actually beengenerated by a user. For example, a sound pattern generated by a userthrough actions such as handclaps is collected by the microphone 15, andsupplied to the PC 10 d as a sound signal. The PC 10 d analyzes thesupplied sound signal to detect a sound pattern, associates the detectedsound pattern with identification information acquired from theidentification image 41 displayed on the medium 40, and registers thesound pattern in the user image DB 230.

In this case, it is preferred that the PC 10 d or the like be used todisplay a message to prompt the registration of a sound pattern and auser interface for instructing the start of registration (start ofrecording). Generating clicking sound at given intervals makes it easierto register a sound pattern.

FIG. 38 is an exemplary flowchart schematically illustrating processingfor displaying an image acquired by the image acquisition device 20according to the second embodiment. In FIG. 38, processing common toFIG. 16 referred to above is denoted by the same reference symbol, and adetailed description is omitted.

Prior to the processing in FIG. 38, a user prepares a medium 40 on whichan identification image 41 based on processing identificationinformation and a sheet 50, draws a predetermined picture on the sheet50, and designates a sound pattern in the sound pattern designationregion 420 arranged in the sheet 50. After that, the user sets the sheet50 on the image acquisition device 20, places the medium 40 in theplacement region 44 on the sheet 50, and causes the image acquisitiondevice 20 to start image acquisition processing. The image acquisitiondevice 20 transmits the acquired image to the PC 10 d.

The PC 10 d receives the image transmitted from the image acquisitiondevice 20, and supplies the received image to the image analysis unit200 d. The image analysis unit 200 d acquires the supplied image (StepS10). At the next Step S210, the image analysis unit 200 b detects ahand-drawing region 43, an attribute setting region 42, and a soundpattern designation region 420 from the supplied image, and extracts auser image, attribute information, and sound pattern information fromthe respective detected regions. The image analysis unit 200 d extractsan identification image from the supplied image, and decodes theextracted identification image to acquire identification information.

At the next Step S12, the registration unit 201 d stores the user image,the attribute information, and the sound pattern information extractedat Step S210 in the user image DB 230 in association with theidentification information, thereby registering the user image. Forexample, the registration unit 201 d stores the sound patterninformation together with the user image and the attribute informationin the third level folder described above with reference to FIG. 23. Bystoring the sound pattern information directly in the second levelfolder, common sound pattern information can be associated with eachuser image stored in each third level folder.

At the next Step S13, the display control unit 210 d sets parameters p₀to p₇ for controlling display of the user image as described above. AtStep S14, the display control unit 210 d sets coordinates in a definedregion for the user image to which the parameters p have been set, andcombines the user image with the background image 110 to generate animage 100. In this manner, the user image is displayed as a user objectin a 3D space defined as a defined region.

FIG. 39 is an exemplary flowchart schematically illustrating user imagedisplay processing based on a sound pattern according to the secondembodiment. Prior to the processing in the flowchart in FIG. 39, a userregisters a user image, attribute information, and sound patterninformation in the user image DB 230 in advance in association withidentification information in accordance with the above-mentionedflowchart in FIG. 38, for example.

For example, the user generates a sound pattern designated in the soundpattern designation region 420 on the sheet 50 toward the microphone 15by handclaps. A sound signal output from the microphone 15 is input tothe PC 10 d, and supplied to the display control unit 210 d via thevoice processing unit 240 as sound information. The display control unit210 d uses the sound pattern detection unit 2108 to analyze the soundpattern on the basis of the supplied sound information (Step S200). Forexample, the sound pattern detection unit 2108 may normalize the levelof the sound signal on the basis of the sound information, and performthreshold determination on the normalized level to detect a soundpattern.

At the next Step S201, the display control unit 210 d searches the userimage DB 230 on the basis of the detected sound pattern, and acquires auser image associated with the sound pattern from the user image DB 230.At the next Step S202, the display control unit 210 d determinesparameters p₀ to p₇ for the user image acquired at Step S201 similarlyto the processing at Step S13 in FIG. 16, and sets the parameters p₀ top₇ for the respective user images.

At the next Step S203, the display control unit 210 d sets coordinatesin a defined region for the user image to which the parameters p havebeen set, and combines the user image with a background image 110 togenerate an image 100. In this manner, each user image associated with asound pattern is displayed as a user object in a 3D space defined as adefined region.

At Step S203, each user object newly appears and is displayed in theimage 100. In this case, the appearance position of the user object canbe determined in advance. For example, the appearance position may beeither the right or left end portion in the image 100.

FIG. 40 is an exemplary flowchart illustrating display processingperformed when a user object for a user image associated with a soundpattern has already been displayed in the image 100 according to thesecond embodiment. In the PC 10 d, the display control unit 210 d startsto collect sound with the microphone 15 (Step S220).

At the next Step S221, the display control unit 210 d determines whethersound has not been detected for a given period. For example, the displaycontrol unit 210 d uses the RAM 1003 as a buffer memory to store thereina sound signal having a length corresponding to an assumed soundpattern, and determines whether sound is present on the basis of thestored sound signal. When the display control unit 210 d determines thatsound has not been detected for a given period (“Yes” at Step S221), thedisplay control unit 210 d returns the processing to Step S221.

The processing at Step S221 is repeatedly executed while the period isshifted little by little. In the buffer memory, a sound signal for agiven period is always stored while the period is shifted little bylittle until the sound collection is finished.

When the display control unit 210 d determines that sound has beendetected within a given period (“No” at Step S221), the display controlunit 210 d advances the processing to Step S222. At Step S222, thedisplay control unit 210 d acquires coordinates in the image 100corresponding to the position at which the sound has been detected.

In the above-mentioned example in FIG. 33, only one microphone 15 isarranged, and hence the position at which sound is detected correspondsto the position of the microphone 15. Without being limited thereto, onemicrophone may be installed at each end of the projection medium 14 tocollect sound in stereo. When sound is collected in stereo, any positionin the horizontal direction of the projection medium 14 can be a sounddetection position.

At the next Step S223, the display control unit 210 d analyzes a soundpattern on the basis of the sound signal stored in the buffer memory. Atthe next Step S224, the display control unit 210 d compares the soundpattern analyzed and acquired at Step S223 with sound patternsregistered in the user image DB 230. At the next Step S225, the displaycontrol unit 210 d determines whether the user image DB 230 has a soundpattern that matches with the sound pattern acquired at Step S223. Whenthe display control unit 210 d determines that the user image DB 230does not have any sound pattern that matches with the sound patternacquired at. Step S223 (“No” at Step S225), the display control unit 210d returns the processing to Step S221.

On the other hand, when the display control unit 210 d determines thatthe user image DB 230 has a sound pattern that matches with the soundpattern acquired at Step S223 (“Yes” at Step S225), the display controlunit 210 d advances the processing to Step S226. At Step S226, thedisplay control unit 210 d acquires a user image associated with thematched sound pattern from the display queue region. At the next StepS227, the display control unit 210 d moves a user object correspondingto the user image acquired from the display queue region toward thecoordinates acquired at Step S222 at which the sound has been detected.The display control unit 210 d returns the processing to Step S221.

Controlling the motion of a user object in accordance with the detectionof sound in this manner enables the motion of the user object to haveinteractiveness.

While the sound pattern is detected on the basis of sound collected bythe microphone 15 in the above, the detection method is not limited tothis example. For example, the sound pattern may be detected with use ofan instrument such as a musical instrument. Examples of the instrumentsinclude an instrument formed by providing a switch to a percussioninstrument such as castanets so that a signal is output when theinstrument is tapped. The signal output from the instrument is suppliedto the PC 10 d. The PC 10 d analyzes the signal supplied from theinstrument to detect a sound pattern. In this case, by includingpositional information on the instrument in the output signal of theinstrument, the control based on coordinates at Step S227 can beperformed. The positional information on the instrument may be set tothe instrument in advance. The positional information on the instrumentmay be detected by the sensor 13.

First Modification of Second Embodiment

Next, a first modification of the second embodiment is described. In thefirst modification of the second embodiment, the control of a userobject by sound in the second embodiment is combined with theacquisition of a user image from the identification image 41 in theabove-mentioned first embodiment.

FIG. 41 illustrates an exemplary configuration of a display systemaccording to the first modification of the second embodiment. In FIG.41, the portions common to FIG. 1 and FIG. 33 referred to above aredenoted by the same reference symbols, and detailed descriptions areomitted. In FIG. 41, the illustration of the image acquisition device 20and the identification image reading device 21 illustrated in FIG. 1 isomitted.

In FIG. 41, in a display system 1 e, an identification image readingdevice 16 is provided near a projection medium 14, desirably near amicrophone 15. The identification image reading device 16 has the samefunction as that of the above-mentioned identification image readingdevice 21. The identification image reading device 16 acquires an imageof the medium 40, reads an identification image 41 displayed on themedium 40 from the acquired image, and supplies the identification image41 to a PC 10 e.

The identification image reading device 16 may be provided as a deviceindependently from the identification image reading device 21, or may beshared with the identification image reading device 21. For the PC 10 e,the configuration of the PC 10 a described above with reference to FIG.12 can be directly applied, and the function described above withreference to FIG. 35 can be directly applied. Thus, the descriptions ofthe configuration and the function of the PC 10 e are omitted. Forexample, the output from the identification image reading device 16 issupplied to a display control unit (corresponding to the display controlunit 210 d in FIG. 35) in the PC 10 e.

Similarly to the DB 12 d in the above-mentioned second embodiment, a DB12 e can store therein a sound pattern and a user image in associationwith each other. More specifically, the DB 12 e includes a user image DB230 and a display image DB 231, and the user image DB 230 stores thereina user image, attribute information, and sound pattern information inassociation with identification information.

FIG. 42 s an exemplary flowchart schematically illustrating processingaccording to the first modification of the second embodiment. Prior tothe processing in the flowchart in FIG. 42, a user registers in advancea user image, attribute information, and sound pattern information inthe user image DB 230 in association with identification information inaccordance with, for example, the above-mentioned flowchart in FIG. 38.

At Step S240, the PC 10 e searches DB 12 e on the basis ofidentification information acquired from an image of the medium 40 readby the identification image reading device 16, and specifies a userimage associated with the identification information.

For example, the user sets the medium 40 on which the identificationimage 41 is displayed on the identification image reading device 16. Inresponse to an instruction from the user's operation, the identificationimage reading device 16 reads the image of the medium 40, and transmitsthe read image to the PC 10 e. The PC 10 e decodes the identificationimage 41 included in the image transmitted from the identification imagereading device 16, and acquires identification information. The PC 10 esearches the user image DB 230 included in the DB 12 e on the basis ofthe acquired identification information, and specifies the user imageassociated with the identification information.

Next, at Step S241 the PC 10 e acquires a sound pattern on the basis ofa sound signal supplied from the microphones 15, and searches the DB 12e on the basis of the acquired sound pattern to specify the user imageassociated with the sound pattern.

For example, the user generates the sound pattern registered in advancein association with the user image toward the microphone 15 byhandclaps. The sound signal output from the microphones 15 is input tothe PC 10 e. As described above for Step S201 in FIG. 39, the PC 10 edetects a sound pattern from the input sound signal. On the basis of thedetected sound pattern, the PC 10 e searches the user image DB 230included in the DB 12 e to specify a user image associated with thesound pattern.

At the next Step S242, the PC 10 e displays, on the basis of theprocessing results at Step S240 and Step S241, the user image specifiedin common to the identification information and the sound pattern on theimage 100 as an object in a 3D space.

For example, the PC 10 e acquires the user image specified in common tothe identification information and the sound pattern from the user imageDB 230 included in the DB 12 e together with corresponding attributeinformation. On the basis of the attribute information, the PC 10 edetermines parameters p₀ to p₇ for the acquired user image similarly tothe processing at Step S13 in FIG. 16. The PC 10 e sets coordinates in adefined region for the user image to which each of the parameters p hasbeen set, and combines the user image with the background image 110 togenerate an image 100. In this manner, each user image specified incommon to the identification information and the sound pattern isdisplayed as a user object in a 3D space defined as the defined region.

An application example of the first modification of the secondembodiment is schematically described. For example, parent and childusers (parent is user A, child is user B) are assumed as users who usethe display system 1 e according to the first modification of the secondembodiment. The user A sets a medium 40 on which an identification image41 is displayed on the identification image reading device 16, andcauses the identification image reading device 16 to read an image ofthe medium 40. It is preferred that the identification image readingdevice 16 be configured to detect that the medium 40 has been set, andautomatically start reading the image.

The user B generates a sound pattern registered in advance near themicrophone 15 by handclaps, for example. In this manner, a user objectrelated to the user image specified in common to identificationinformation based on the identification image 41 and the sound patternappears on the image 100.

This configuration enables a user A who is a parent to watch a user Bwho is a child, and enables the display system 1 e to specify a userimage associated with identification information based on anidentification image 41 on a medium 40. Thus, for example, the userkeeps rhythm with handclaps in front of the microphone 15, the displaysystem 1 e can specify a user image associated with the rhythm andassociated with identification information, and display the specifieduser image in the image 100 as a user object.

For example, a part of the image 100 (either the right or left end ispreferred) is set as a re-entry region for a user object in advance. Inorder to cause a user object that has exited from the image 100, forexample, a user object that has moved out of the image 100 to appear inthe image 100 again, a user A sets a medium 40 on which anidentification image 41 is displayed on the identification image readingdevice 16, and a user B is requested to perform an operation of clappingwith the rhythm registered in advance. In this manner, a sense of rhythmof the user B can be improved while the game enjoyment is given.

Second Modification of Second Embodiment

It is more preferred, that the display system 1 e according to the firstmodification of the second embodiment illustrated in FIG. 41 be providedwith a PC configured to present an operation guidance near theidentification image reading device 16. FIG. 43 illustrates an exemplaryconfiguration of a display system provided with an operation guidance PCaccording to a second modification of the second embodiment. In FIG. 43,the portions common to FIG. 1 and FIG. 41 referred to above are denotedby the same reference symbols, and detailed descriptions are omitted. InFIG. 43, the illustration of the image acquisition device 20 and theidentification image reading device 21 illustrated in FIG. 1 is omitted.

In FIG. 43, a display system 1 e′ is obtained by adding a PC 70 to thedisplay system 1 e in FIG. 41. Similarly to the PC 10 e, the PC 70 canacquire identification information on the basis of an image output fromthe identification image reading device 16. In addition thereto, the PC70 can acquire a set of a user image, attribute information, and soundpattern information associated with the identification information froma user image DB 230 included in a DB 12 e.

For the PC 70, the configuration of the PC 10 a described above withreference to FIG. 12 can be directly applied, and hence the descriptionof the configuration of the PC 70 is omitted.

The PC 70 can acquire, from the DB 12 e, a user image associated withidentification information based on an image output from theidentification image reading device 16, and display the acquired userimage on a monitor included in the PC 70. In this case, the PC 70 maygenerate thumbnail images by reducing the size of the acquired userimage, and display a list of the thumbnail images on the monitor.

For example, in the case where a name writing region for writing a nameof an image in the hand-drawing region 43 is provided on the sheet 50,name information written in the name writing region may be displayed ona monitor. For example, the PC 10 e further extracts an image in thename writing region from the image of the sheet 50 acquired from theimage acquisition device 20, and stores the extracted image in the userimage DB 230 included in the DB 12 e as name information in associationwith identification information together with the user image, theattribute information, and the sound pattern information. The PC 10 emay use a character string obtained by performing character recognitionon the image in the name writing region as name information. The PC 70acquires, from the DB 12 e, name information associated withidentification information based on the image output from theidentification image reading device 16, and displays a list of nameinformation on the monitor.

The PC 70 can acquire, from the DB 12 e, sound pattern informationcorresponding to a thumbnail image or name information designated fromthe above-mentioned list of thumbnail images or name information byuser's operation, and present the acquired sound pattern information toa user as a guidance for a sound pattern to be generated. The guidancefor the sound pattern may be presented to the user by being displayed ona monitor as visual information such as a musical score, or may bepresented to the user as acoustic information by generating sound inaccordance with the sound pattern.

For another example, the position of the user B may be detected by thesensor 13, and a user object for a user image associated withidentification information in an image 100 may be moved in accordancewith the detected position. In one example, when the user B moves fromside to side in front of the image 100 (projection medium 14), userobjects for a user image associated with identification information movein a row in accordance with the movement of the user B. An instrumentsuch as castanets provided with a switch as described above may be used.In this case, the position of the instrument may be detected byproviding a position detection mechanism using a gyro sensor inside theinstrument, or a sensor configured to detect the position of theinstrument may be additionally provided.

While each of the above-mentioned embodiments is a preferred embodimentof the present invention, the present invention is not limited theretoand can be variously modified in the range not departing from the gistof the present invention.

The embodiments exhibit an advantageous effect that the motion of anobject in a three-dimensional space can be reflected to operation of auser image displayed in a screen.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example, atleast one element of different illustrative and exemplary embodimentsherein may be combined with each other or substituted for each otherwithin the scope of this disclosure and appended claims. Further,features of components of the embodiments, such as the number, theposition, and the shape are not limited the embodiments and thus may bepreferably set. It is therefore to be understood that within the scopeof the appended claims, the disclosure of the present invention may bepracticed otherwise than as specifically described herein.

The method steps, processes, or operations described herein are not tobe construed as necessarily requiring their performance in theparticular order discussed or illustrated, unless specificallyidentified as an order of performance or clearly identified through thecontext. It is also to be understood that additional or alternativesteps may be employed.

Further, any of the above-described apparatus, devices or units can beimplemented as a hardware apparatus, such as a special-purpose circuitor device, or as a hardware/software combination, such as a processorexecuting a software program.

Further, as described above, any one of the above-described and othermethods of the present invention may be embodied in the form of acomputer program stored in any kind of storage medium. Examples ofstorage mediums include, but are not limited to, flexible disk, harddisk, optical discs, magneto-optical discs, magnetic tapes, nonvolatilememory, semiconductor memory, read-only-memory (ROM), etc.

Alternatively, any one of the above-described and other methods of thepresent invention may be implemented by an application specificintegrated circuit (ASIC), a digital signal processor (DSP) or a fieldprogrammable gate array (FPGA), prepared by interconnecting anappropriate network of conventional component circuits or by acombination thereof with one or more conventional general purposemicroprocessors or signal processors programmed accordingly.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), digital signal processor (DSP), fieldprogrammable gate array (FPGA) and conventional circuit componentsarranged to perform the recited functions.

What is claimed is:
 1. A display apparatus, comprising: an imageacquisition unit configured to acquire an image including a drawingregion drawn by a user; an image extraction unit configured to extract,from the acquired image, a first image being an image in the drawingregion; a registration unit configured to register attribute informationindicating attributes that is set with respect to the extracted firstimage and is used for controlling of moving the first image on a screen;a display control unit configured to control display on the screen; acoordinate generation unit configured to generate, based on a detectionresult of a detection unit configured to detect a position of an objectin a three-dimensional space, coordinates of the object in the screen;and a motion detection unit configured to detect a motion of the objectbased on the coordinates, wherein the display control unit is configuredto further display, when the motion is detected by the motion detectionunit, a second image on the screen based on the coordinatescorresponding to the detected motion, and change the display of thefirst image to which attribute information of a certain attribute amongthe attributes of the first image displayed on the screen is set.
 2. Thedisplay apparatus according to claim 1, wherein the display control unitis configured to move, when the motion is detected by the motiondetection unit, a position of the first image toward a position of thesecond image.
 3. The display apparatus according to claim 1, wherein thedisplay control unit is configured to execute, when the second image isdisplayed on the screen, an emphasis processing for emphasizing that thesecond image is displayed.
 4. The display apparatus according to claim3, wherein the display control unit is configured to further display athird image in vicinity of the second image as the emphasis processing.5. The display apparatus according to claim 3, wherein the displaycontrol unit is configured to determine whether at least a part of thesecond image overlaps with the first image based on the coordinates, andwhen determining that at least a part of the second image overlaps withthe first image, change the display of the first image as the emphasisprocessing.
 6. The display apparatus according to claim 1, wherein thedisplay control unit is configured to arrange the first image in aregion having an attribute corresponding to an attribute of the firstimage within the screen.
 7. The display apparatus according to claim 1,further comprising a display image acquisition unit configured toacquire display contents on the screen as a capture image, wherein thedisplay control unit is configured to instruct the display imageacquisition unit to acquire the capture image when the coordinates areincluded in a predetermined region in the screen.
 8. The displayapparatus according to claim 1, wherein the display control unit isconfigured to change the display of the first image when the detectionresult is not obtained for a given period.
 9. The display apparatusaccording to claim 1, wherein the first image is a display imagepresented by a user, and the second image is an image stored in astorage unit in advance.
 10. A display control method, comprising:acquiring an image including a drawing region drawn by a user;extracting, from the acquired image, a first image being an image in thedrawing region; registering attribute information indicating attributesthat is set with respect to the extracted first image and is used forcontrolling of moving the first image on a screen; controlling displayon the screen; generating, based on a detection result of a detectionunit configured to detect a position of an object in a three-dimensionalspace, coordinates of the object in the screen; and detecting a motionof the object based on the coordinates, wherein the displaying comprisesdisplaying, when the motion is detected at the motion detection step,further displaying a second image on the screen based on the coordinatescorresponding to the detected motion, and changing the display of thefirst image to which attribute information of a certain attribute amongthe attributes of the first image displayed on the screen is set.
 11. Adisplay system, comprising: an image acquisition unit configured toacquire an image including a drawing region drawn by a user; an imageextraction unit configured to extract the first image from the imageacquired by the image acquisition unit, the first image being an imagein the drawing region; a registration unit configured to registerattribute information indicating attributes that is set with respect tothe extracted first image and is used for controlling of moving thefirst image on a screen; a storage unit configured to store therein asecond image; a display control unit configured to control display onthe screen; a coordinate generation unit configured to generate, basedon a detection result of a detection unit configured to detect aposition of an object in a three-dimensional space, coordinates of theobject in the screen; and a motion detection unit configured to detect amotion of the object based on the coordinates, wherein the displaycontrol unit is configured to display the first image on the screen, andwhen the motion is detected by the motion detection unit, furtherdisplay the second image on the screen based on the coordinatescorresponding to the detected motion, and changes the display of thefirst image to which attribute information of a certain attribute amongthe attributes of the first image displayed on the screen is set.